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"We  are  like  dwarfs  seated  on  the  shoulders  of  giants; 
we  see  more  things  than  the  ancients  and  things  more 
distant,  but  this  is  due  neither  to  the  sharpness  of  our 
own  sight,  nor  to  the  greatness  of  our  own  stature, 
but  because  we  are  raised  and  borne  aloft  on  that 
giant  mass.” 


Bernard  of  Chartres 


-  II 


ABSTRACT 


A  syntax-directed  model  is  presented  which  is  able  to  recognize  and  gen¬ 
erate  two-dimensional  pictures  while  allowing  a  high  degree  of  man  machine 
interaction.  Starting  with  a  field  of  points  representing  the  picture,  a  string  of 
symbols  providing  a  structural  description  of  the  picture  is  produced  by  the 
syntactic  component.  The  structural  description,  composed  of  higher  level 
primitives  (e.g.,  geometric  symbols  such  as  triangles  and  rectangle, s)  and  syn¬ 
tactic  relations  which  exist  between  the  primitives,  is  operated  upon  by  the 
semantic  component  to  provide  a  semantic  interpretation  for  the  picture.  The 
syntactic  component  consists  of  a  lexicon,  a  modified  context-sensitive  phrase 
structure  grammar,  and  a  set  of  transformation  rules.  The  semantic  component 
consists  of  a  set  of  heuristics  to  abstract  the  picture  and  a  modified  context- 
sensitive  phrase  structure  grammar  which  allows  contextual  restrictions  to  be 
applied  to  combinations  of  constituents  existing  at  different  levels  of  the  syntax 
tree  which  syntactically  describes  the  picture.  Various  aspects  of  the  model 
have  been  programmed  on  the  LINC  (a  small  digital  computer),  the  IBM  3(i0  '50, 
and  the  IBM  7072. 
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SAP:  A  MODEL  FOR  THE  SYNTACTIC  ANALYSIS  OF  PICTURES 


1.  INTRODUCTION 


The  syntactic  analysis  of  natural  language  has  become  a  well-established  technique  in 
the  field  of  linguistics.  Whether  a  similar  analysis  can  successfully  be  applied  to  pictures 
remains  to  be  seen.  This  report  is  an  effort  to  consider  some  of  the  problems  which  arise  in  the 
syntax-directed  analysis  of  pictorial  data,  While  it  is  seen  that  the  syntactic-semantic  approach 
may  be  used  in  both  the  analysis  of  linguistic  and  picto.ial  data,  many  of  the  formulations 
developed  in  linguistics  should  not  be  expected  to  carry  ove  to  pictures. 

A  most  satisfactory  situation  would  be  one  in  which  tnc  computer  has  the  ability  to  perform 
as  well  as  a  human  in  the  field  of  pattern  recognition.  Wh.'e  there  arc  industrial  problems  which 
require  the  machine  to  make  discriminations  beyond  that  which  is  humanly  possible,  a  pattern 
recognizer  which  could  recognize  checkbook  signatures,  postage  stamps,  or  airplane  silhouettes 
would  be  no  small  achievement.  Thus,  the  field  of  psychology  may  be  an  information  source 
for  pattern  recognition  in  terms  of  such  work  as  has  been  done  by  the  Gestalt  psychologists. 
However,  a  distinction  must  be  maintained  between  a  model  which  simulates  human  intelligence 
and  a  model  which  can  provide  the  same  results  as  human  intelligence. 

In  developing  this  report,  the  pictorial  data  was  restricted  to  straight  line  figures  which 
have  the  semblance  of  cartoon-like  drawings  with  no  field  of  depth.  Thus,  the  pictures  may  be 
considered  to  represent  three-dimensional  objects  projected  onto  a  plane  with  the  point  of  view 
of  the  object  being  perpendicular  to  any  one  of  its  major  axes.  In  the  following  chapters  there 
is  no  distinction  made  between  the  terms  graphics  and  pictures  implying  that  the  digitized 
points  of  the  pictorial  data  can  be  the  result  of  drawing  on  a  cathode-ray  tube  with  a  light  pen 
or  scanning  a  hard  copy  photograph. 

As  a  formal  definition,  the  picture  which  is  presented  to  the  computer  is  called  a  scene. 
The  scene,  in  turn,  is  composed  of  figures.  The  figures  are  built  up  from  constituents  called 
primitives.  The  higher  level  primitives  for  the  examples  used  in  this  report  are  geometric 
symbols^  such  as  triungies,  circle,  etc,,  and  the  figures  are  houses,  trees,  etc.  The  primitives 
are  combined  to  form  the  figures  by  syntactic  relations.  The  syntactic  relations  used  in  the 
present  examples  are  on  top  of,  etc. 

This  report  presents  a  model  which  provides  an  approach  to  the  syntactic  analysis  of 
pictures.  As  indicated  earlier,  the  pictures  are  two-dimensional  patterns  which  have  significance 
in  the  real-world.  The  word  analyzer  is  meant  to  indicate  the  ability  to  both  recognize  and 
generate  pictures.  Syntactic  analysis  indicates  that  a  structural  description  is  obtained, 
describing  the  topological  features  of  the  picture.  It  is  on  the  basis  of  the  structural  de.  ..rip- 
tion  that  the  pattern  recognition  (called  semantic  interpretation)  is  accomplished, 

The  model.  SAP  (Syntactic  /Inalyzer  of  Pictures),  was  developed  hased  on  an  interest 
in  the  general  methodology  and  philosophy  of  syntax-directed  analysis,  and  as  such,  provides 
an  overall  view  of  the  problem.  The  model,  as  described  in  this  report,  is  composed  of  two 
major  components,  a  syntactic  component  and  a  semantic  component.  A  syntax-directed  meta¬ 
language  to  facilitate  man/machine  interaction  is  also  described  in  detail, 


An  outline  of  SAP  and  the  ability  of  the  user  to  interact  is  represented  in  Figure  1. 
The  set  of  two-dimensional  pictures  acceptable  to  SA P  constitutes  a  language  L.  The  syntactic 
component  of  SAP  accepts  a  picture  L,  and  translates  it  to  a  one-dimensional  string  L*,. 
This  one-dimensional  string  L*,  is  a  structural  description  of  the  picture  L,.  The  set  of  all 
structural  descriptions  of  L  constitutes  a  language  L*.  The  string  L*,  is  sent  to  the  semantic 
component  to  allow  an  identification  or  Label,  to  be  applied  to  picture  L,.  This  process  is 
represented  by  the  solid  lines  in  Figure  1.  The  inverse  process  is  the  insertion  of  a  Label, 
into  SAP,  whereby  a  picture  L,  is  generated.  This  is  represented  by  the  dashed  lines  in 
Figure  1.  It  is  interesting  to  note  that  the  use  of  L,  or  Label,  as  input  does  not  uniquely 

determine  the  other  as  output. 

The  user  is  able  to  present  SAP  with  a  syntactic  string  L*,.  A  syntactic  metalanguage 
L**  allows  the  user  to  present  SAP  with  only  well-formed  strings.  A  structural  description 
created  by  the  user  can  be  sent  to  the  semantic  component  (solid  line)  to  receive  a  Label(  or 
sent  to  the  syntactic  component  (dashed  line)  to  generate  a  picture. 

The  second  chapter  describes  and  defines  the  basic  concepts  which  are  taken  from  the 
fields  of  linguistics  and  computer  science.  The  nature  of  graphics  is  discussed  as  is  the 
difference  between  problems  of  the  syntactic  analysis  of  a  natural  language  and  problems  of 
the  syntactic  analysis  of  pictures. 

The  third  chapter  provides  a  survey  and  contrast  of  the  various  syntax-directed  pattern 
recognition  systems  which  have  been  described  in  the  literature. 

The  fourth  chapter  is  a  formal  presentation  of  a  syntax-directed  language  which  has  been 
designed  to  enable  a  user  to  describe  by  a  one-dimensional  string  of  symbols  the  two-dimen¬ 
sional  type  of  pictorial  data  that  SAP  is  able  to  consider.  The  strings  are  composed  of  symbols 
representing  the  geometric  primitives  and  syntactic  relations  which  comprise  the  pictorial  data. 

The  fifth  chapter  describes  the  syntactic  component  of  SAP.  Rules  are  presented  to 
combine  a  set  of  digitized  points  into  lines  and  the  lines  into  geometric  symbols.  A  lexicon 
is  used  to  obtain  the  syntactic  relations  between  the  geometric  symbols.  A  syntax  tree  is 
formed  and  from  this  a  linear  string  is  derived  to  represent  the  syntactic  structure  of  the  picture. 

The  sixth  chapter  is  a  description  of  the  semantic  component  of  SAP.  The  syntactic 
string  is  abstracted  to  obtain  individual  figures  and  their  basic  characteristics,  or  Gestalt 
features.  A  semantic  analysis  using  context  sensitive  rules  attempts  to  identify  the  figure  on 
the  basis  of  its  syntactic  structure.  Unidentified  figures  are  then  identified  by  context  sen¬ 
sitive  rules  in  terms  of  the  syntactic  structure  of  the  scene. 

The  seventh  chapter  provides  a  discussion  of  possible  further  extensions  of  SAP  and 
syntax-directed  pattern  analysis  models  in  general.  Also  offered  is  a  discussion  of  the 
advantages  and  disadvantages  of  such  a  model. 

The  eighth  chapter  contains  a  summary  and  conclusions  of  the  work  presented  in  this 

report. 

In  Appendix  9.1  is  contained  a  description  of  the  implementation  of  SAP  on  the  L1NC1 
in  LAP62,  the  IBM  360/50  in  LISP  1.53-4,  and  the  IBM  7072  in  FORTRAN.  Appendix  9.2. 
contains  a  listing  of  the  grammars  and  transformation  rules  presented  throughout  the  report. 

To  provide  a  basis  for  the  discussion  of  the  following  chapter,  it  will  be  assumed  that 
graphics  (or  two-dimensional  pictures)  is  some  form  of  language.  The  terms  graphics  and  pictures 
will  be  used  interchangeably.  Whether  the  two-dimensional  picture  language  should  be  considered 
a  natural,  artificial,  or  other  type  of  language  is  unclear.  While  no  claim  is  made  for  a  picture 
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language  as  being  a  form  of  natural  language,  to  understand  better  the  nature  of  at  least  two- 
dimensional  pictures,  they  will  be  described  in  terms  of  some  of  the  considerations  given  to 
English  by  a  structural  or  syntax-directed  approach  to  natural  languages. 

The  choice  of  geometric  symbols  as  the  higher  level  primitives  was  made  partially  as  a 
matter  of  convenience  and  also  on  the  basis  of  the  desired  graphic  input.  It  was  required  that 
the  scenes  be  rich  enough  in  contextual  information  to  allow  our  model  sufficient  opportunity  to 
be  tried  but  not  overwhelmingly  complex  as  to  cause  an  excessive  number  of  side  tracking 
problems.  Thus,  it  was  decided  to  work  with  out-door  type  scenes  as  found  in  the  country  or 
city.  In  addition,  it  was  thought  best  to  begin  work  in  two  dimensions.  In  view  of  this,  a 
basic  set  of  geometric  symbols  were  chosen.  It  was  rather  surprising  the  large  number  of 
sophisticated  figures  which  could  be  drawn  from  the  small  set  of  geometric  primitives.  A  page 
ot  these  scenes  is  found  in  Figure  2.  However,  it  should  be  noted  that  the  pattern  recognition 
ol  these  particular  scenic  figures  is  irrelevant  to  the  goal  of  indicating  a  general  approach  to 
the  syntactic  analysis  of  pictorial  data. 

The  syntactic  string  which  is  processed  by  the  pattern  recognizer  may  be  considered  a 
data  structure.  This  data  structure  allows  not  only  the  syntax  of  the  graphic  input  to  be  con¬ 
cisely  represented,  but  also  allows  the  semantics  of  the  scene  to  be  obtainable.  The  semantics 
of  the  scene  are  the  various  meanings  or  recognitions  that  can  be  made  in  conjunction  with  the 
figures  of  the  scene,  though  other  semantic  levels  of  pictorial  expressions  could  be  defined. 
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2.  BASIC  CONCEPT?  AND  CONSIDERATIONS 


2.1  SYNTAX-DIRECTED  PATTERN  RECOGNITION 


As  indicated  in  : he  introduction,  syntactic  analysis  is  meant  to  imply  both  the  generation 
and  the  recognition  of  pictures.  However,  strictly  speaking  syntax-directed  analysis  refers  to 
the  recognition  of  patterns. 

As  indicated  by  Unger,  there  is  a  distinction  between  pattern  recognition  and  pattern 
detection. 


Pattern  deteetion  consists  of  examining  an  arbitrary  set  of  figures  and 
selecting  those  having  some  specified  form.  Pattern  recognition  con¬ 
sists  of  identifying  a  given  figure  which  is  known  to  belong  to  one  of 
a  finite  set  of  classes. 

Patterns  can  be  considered  as  a  large  number  of  ordered  discrete  points.  By  giving  an 
identification  or  label  to  a  pattern  essentially  a  many-to-one  mapping  is  being  performed. 
Accepting  pattern  recognition  to  be  a  many-to-one  mapping,  the  difficulty  in  pattern  recog¬ 
nition  becomes  one  of  determining  what  operations  will  perform  this  mapping.  The  operation 
which  performs  this  mapping  must  obtain  a  set  of  measurements  (n-tuples)  which  characterize 
the  pattern.  Thus,  a  major  problem  of  any  pattern  recognition  model  is  the  choice  of  measure¬ 
ments  with  which  it  attempts  to  characterize  the  patterns  which  it  wishes  to  name. 

1  he  syntax-directed  method  of  pattern  recognition  essentially  analyzes  the  patterns  for 
connectivity  and  topological  features.  By  considering  the  topology  and  geometry  of  the  patterns, 
sets  of  n-tuples  are  formed,  which  are  geometrically  and  topologically  related.  The  formation 
of  these  sets  is  accomplished  by  a  grammar.  The  grammar  may  be  providing  a  syntactic  or 
semantic  analysis.  Hence,  a  syntax-directed  pattern  recognition  model  may  be  represented  as 
a  set  of  grammars  which  serve  as  input  to  the  computer  along  with  the  input  pattern,  which  is 
to  be  identified.  The  computer  then  serves  as  a  translator  which  operates  according  to  the 
grammar  rules  to  translate  (map)  from  a  picture  L,  to  its  Label,. 

I  he  input  of  the  grammars  for  the  syntactic  and  semantic  analyses  constitutes  providing 
a  syntactic  component  and  a  semantic  component  to  the  computer.  This  is  shown  in  Figure  3. 
In  terms  of  SAP,  it  is  actually  the  semantic  component  which  performs  the  recognition  or 
labeling  of  the  picture.  The  syntactic  component  provides  a  structural  description  of  the 
picture  which  allows  this  recognition  to  take  place. 


2.2  THE  PHRASE  STRUCTURE  GRAMMAR 


A  graphic  or  pictorial  language  L  is  considered  to  be  a  subset  of  the  set  A*  of  all  finite 
arrays  of  symbols  from  an  alphabet  A.  The  language  L  is  generated  by  the  set  of  alpha¬ 
betic  symbols  A  and  a  set  of  rules  for  combtning  these  symbols  into  a  hierarchy  of  consti¬ 
tuents.  The  alphabet  plus  the  set  of  rules  is  called  a  grammar.  Essentially,  a  grammar 
provides  a  description  to  account  for  observed  patterns,  and  thus  may  be  considered  an  ab¬ 
straction  of  these  common  patterns.  A  theory  of  graphics  would  be  required  to  choose  an 
adequate  grammar  on  the  basis  of  given  graphic  data.  While  criterion  used  in  linguistics, 
such  as  Chomsky’s4  descriptive  and  explanatory  levels  of  adequacy  might  be  valuable  it 
certainly  would  be  premature  to  propose  such  criterion  for  a  starting  point  in  a  theory  of  graphics. 


r'ignrc  3.  The  Syntnx-Directcd  Model 
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I'lie  type  of  grammar  which  is  used  to  perform  the-  analyses  of  the  graphic  or  pictorial 
data  is  called  a  phrase  structure  grammar  (PSG).  While  the  components  of  SAP  use  modifi 
cations  of  the  standard  PSCi,  a  definition  of  a  PSG  grammar  is  as  follows: 

Essentially  a  phrase  structure  grammar,  K.  is  a  finite  set  of  productions  of  the  torm, 
i>— where, 

(i)  i>  is  a  nonnull  string  of  symbols  of  the  vocabulary  V, 

(ii)  6  is  a  single  symbol  of  the  type  called  a  nonterminal  symbol 
of  the  vocabulary  V. 

(iii)  I  here  is  one  <8  called  the  goal,  S,  where  the  goal  is  never  a 
member  of  a  6  string. 

The  vocabulary  V  is  the  union  of  the  set  A  of  alphabetic  symbols  called  terminals 
symbols  and  a  set  N  of  syntactic  metalanguage  symbols  used  for  defining  the  lan¬ 
guage  L  and  which  are  called  nonterminal  symbols. 

A  phrase  structure  grammar  defines  a  language  by  forming  derivatives  of  the  goal,  S,  If  in 
applying  a  production  rule  6 — >  t//,  the  string  ft  is  said  to  be  a  direct  derivative  of  the  string 
a  (a  y  ft)  if  there  are  strings  y  and  8  such  that  a=yc6S  and  ft=yif/8.  The  strings  y  and  8  may  be 
null  in  the  case  that  the  production  rules  are  context  free,  otherwise  the  rules  are.  considered 
context  sensitive.  The  operation  a*)>ft  is  defined  as  the  case  where  there  exist  strings  an, 

“• . “i  !'uc*1  t*lat  n=o0>  "0  =7(!1  .••••  «i_l  and  a j  ft.  In  this  case  ft  is  called  a  derivative  of  a 

and  the  sequence,  *=*(>#«; . «j_i  ^u-^ft  is  called  a  derivation  of  ft  and  «. 

The  derivatives  of  the  goal  are  called  sentential  forms.  Those  sentential  forms  which 
consist  only  of  terminal  symbols  are  called  the  set  of  sentences  (for  pictorial  data,  the  sentences 
are  called  scenes)  and  it  is  the  set  of  scenes  which  comprises  the  language  1..  That  is, 

L  —  |  s  S  s  and  s  f  A*} 


2.3  PARSING  STRATEGIES 

I  he  derivations  of  the  scenes  of  L  from  the  goal  S  are  often  represented  by  tree  structures. 
The  derivations  provide  a  structural  description  (or  parsing)  of  the  scenes.  The  tree  structures, 
called  syntax  trees,  explicitly  represent  the  structural  descriptions  or  parsings. 

The  derivations  of  the  sentences  from  the  goal  S  by  means  of  the  PSG  use  basically  one 
ot  two  algorithms  or  parsing  strategies.  These  are  called  top-down  and  bottom  up  strategies. 

1.  The  top-down  strategy  is  completely  goal  oriented.  The  main  goal  S  is 
chosen  first.  I  his  goal  chooses  a  set  ot  subgoals.  The  subgoals  hope  to 
find  a  derivation  of  the  scenes  from  S  by  substituting  the  right  hand  side 
of  the  production  rules  in  the  place  of  the  subgoals.  The  substitution 
forms  a  new  set  of  subgoals.  Thus,  each  subgoal  in  turn  chooses  a  set 
of  subgoals.  If  a  subgoal  fails  its  task  it  is  rejected  and  a  new  subgoal 
replaces  it.  It  is  hoped  that  the  subgoals  will  eventually  reach  the  ter¬ 
minal  string.  The  top-down  strategy  causes  infinite  looping  if  the  strings 
are  analyzed  from  left-to-right  and  a  left  recursions  rule  occurs,  that  is 
$i  0i|/,i  •  The  parsers  which  use  a  top-down  strategy  are  sometimes 
called  predictive,  since  at  each  step  they  attempt  to  predict  the  subgoals 
to  be  used  to  reach  the  terminal  string. 
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2.  The  bottom  up  strategy  has  only  implicitly  the  long  range  goal,  S.  The 
strategy  is  essentially  an  attempt  to  substitute  substrings  which  are  the 
right  hand  side  of  production  rules  by  their  corresponding  left  hand  side. 

In  this  manner,  it  is  hoped  that  eventually  the  single  symbol  S,  will  be 
reached. 

Both  parsing  strategies  are  used  in  SAP.  The  syntactic  component  uses  a  bottom-up  strategy 
to  obtain  a  structural  description  of  the  pictorial  input.  The  semantic  component  uses  a  right 
to-left  modified  bottom-up  parsing  to  abstract  the  structural  description  provided  by  the  syntactic 
component.  The  semantic  component  also  uses  a  modified  top-down  strategy  to  assign  a  semantic 
interpretation  to  the  figures  of  the  scene  and  a  bottom-up  strategy  to  assign  a  semantic  inter¬ 
pretation  to  the  pictorial  scene. 

2.4  THE  NATURE  OF  PICTORIAL  DATA 

One  aspect  of  pictures  which  should  be  mentioned  is  the  presence  of  pictorial  universals. 
though  as  Hockett7  points  out.  we  do  not  want  to  invent  language  universals  but  discover  them. 
Some  of  these  universals  are: 

(i)  Arbitrary  configurations  of  pictorial  data  (figures)  can  be  created  at  will. 

(ii)  The  newly  created  figures  can  be  considered  discretely  defined  messages. 

(iii)  The  figures  may  be  assigned  meaning  independent  of  any  physical  or 
geometric  form  of  the  figures  and  also  independent  of  the  spatio-temporal 
coordinates  of  the  figure. 

( i v )  For  any  non  trivial  graphic  language,  there  is  the  possibility  of  ambiguous 
and  anomalous  pictoiial  data. 

As  defined  by  Chomsky.8  one  should  note  the  difference  between  a  competence  and  a 
performance  model  of  the  syntax  of  the  language.  Competence  is  to  be  considered  the  viewer's 
knowledge  of  graphics  while  performance  is  the  viewer’s  actual  use  of  this  knowledge.  It  is 
the  ideal  viewer's  intrinsic  competence  which  is  represented  by  a  grammar.  The  grammar  assigns 
to  each  figure  a  structural  description  which  indicates  how  this  figure  is  grammatically  under¬ 
stood  by  the  ideal  viewer.  The  acceptability  of  a  figure  refers  to  the  performance  model  and 
based  on  such  factors  us  memory  limitation  it  is  not  of  present  interest.  The  competence 
model  is  concerned  with  the  grammaticalness  of  a  figure. 

A  pictorial  message  goes  through  the  same  encoding  and  decoding  processes  as  a  linguistic 
message.  Thus,  an  individual  encodes  a  graphic  message  by  creating  it  in  such  a  manner  that 
the  message  can  be  visually  perceived.  The  graphic  message  is  then  decoded  by  either  the 
original  sender  or  another  viewer  when  an  attempt  is  made  to  understand  the  message.  The 
message  can  be  a  single  alphabetic  symbol  called  a  primitive,  a  structure  composed  of  several 
primitives  called  a  figure,  or  a  structure  composed  of  one  or  more  figures  called  a  scene. 

The  encoding  and  decoding  of  a  graphic  message  is  accomplished  by  syntactic  and 
semantic  components.  It  is  the  decoding  operation,  performed  by  the  syntactic  and  semantic 
components,  that  is  generally  considered  syntax-directed  pattern  recognition.  The  syntactic 
component  parses  the  graphic  data.  The  parsed  graphic  data  is  represented  by  a  structural  des¬ 
cription  (SB)  which  indicates  the  primitives  comprising  the  graphic  data  and  the  syntactic  re¬ 
lations  between  the  primitives.  The  semantic  component  is  highly  dependent  on  the  syntactic- 
component  and  probably  should  be  called  the  semantic-syntactic  component.  The  semantic  com¬ 
ponent  accepts  the  structural  description  as  input  and  provides  a  semantic  interpretation  (SI)  to 
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the  graphic  data.  This  semantic  interpretation  indicates  whether  the  data  is  recognized,  ambi 
guous,  or  anomalous. 


A  picture  L.  may  actually  have  more  than  one  structural  description.  Thus,  L*,  repre¬ 
sents  the  set  of  structural  descriptions  of  L..  A  member  of  the  set  L.*j,  structural  description 
j  of  picture  L.,  is  represented  as  L*|,. 

Similarly,  picture  L.  may  have  more  than  one  semantic  interpretation,  in  which  case  the 
picture  is  considered  ambiguous.  Thus,  Label,  represents  the  set  of  semantic  inierpre tations 
assigned  to  picture  L.,  A  particular  identification  or  label  of  L,  is  represented  as  Label).. 

The  decoding  operation  can  be  represented  by  the  following  equations: 

SD(l. .  )  =  L*. 

I  I 

where,  L*|  =  |L*|j|  and  if  for  L*jj, 

(i)  max  j  =  0,  then  L;  is  not  well-formed, 

(ii)  max  j  =•  1.  then  L.  has  a  single  structural  description. 

(iii)  max  j  1,  then  L.  has  multiple  structural  descriptions  (multiple  parsings). 
SKL* ,)  =  Label  j 

where,  Label. =  L  | L abe l ( ;  j-  and  if  for  Label  ( j , 

(i)  max  j  =  0,  then  Lf  is  anomalous. 

(ii)  max  j  =  1,  then  L.  is  singularly  identified. 

(iii)  max  j  >  1,  then  L.  is  ambiguous, 

In  the  remainder  of  the  report,  the  subscript  j  will  be  left  off  unless  a  reference  is  to  be 
made  to  a  particular  structural  description  or  identification. 

The  graphic  data  or  picture  may  also  be  considered  a  geometric  graph.  A  geometric  graph 

in  two-dimensions  is  a  set  V  =  j  v.  ^  of  points  and  a  set  E  -•=  -j  e (  }  o f  simple  curves  satis  ¬ 

fying  the  following  conditions: 

(i)  Every  continuous,  non-self-intersecting  curve  in  E  whose  end  points 
coincide  contains  exactly  one  point  of  V. 

(ii)  Every  continuous,  non-self-intersecting  curve  in  E  which  joins  two 
distinct  points  contains  precisely  two  points  of  V,  and  these  agree 
with  its  end  points. 

(iii)  The  curves  of  E  have  no  common  points,  except  for  points  of  V. 

Thus, a  geometric  graph  is  a  geometric  configuration  or  structure,  in  this  case  in  two-dimensions, 
which  consists  of  a  set  of  points  interconnected  by  a  set  of  nonintersecting  continuous  curves. 
An  example  is  shown  on  the  following  page.  The  fact  that  a  figure  can  be  considered  a  geometric 
graph  will  be  used  in  Chapter  Five. 


2.5  LINGUISTICS  DATA  VERSUS  PICTORIAL  DATA 

While  the  application  of  linguistics  to  graphics  will  be  carried  through  the  remainder  of 
the  report,  it  is  worthwhile  to  note  some  major  difficulties  in  syntax-directed  pattern  recognition 

which,  for  the  most  part,  do  not  appear  in  linguistics, 

(1)  A  figure  can  be  parsed  from  almost  any  direction,  where  theoretically 
each  parsing  readily  can  provide  the  same  information,  While  tins  in 
itself  is  not  a  problem,  since  the  viewer  can  always  approach  the  figure 
from  one  standard  direction  if  necessary,  this  flexibility  indicates  that 
restrictions  must  be  considered  because  of  the  astronomically  large  num¬ 
ber  of  parsings  of  only  a  slightly  complex  picture.  As  indicated  earlier, 
the  set  of  parsings  of  picture  L.  is  represented  by  l.*j,  where  a  parti¬ 
cular  parsing  j  of  picture  L,  is 

(2)  The  syntactic  relations  of  a  grammar  have  a  hierarchy  which  limits  the 
number  of  possible  parsings  of  a  figure  into  its  primitives,  However, 
for  the  restricted  pictorial  data  being  considered  (elimination  of  depth) 
any  correct  parsing  of  the  figure  provides  essentially  the  same  structural 
description  of  the  figure  and  hence,  ideally,  the  same  semantic  inter¬ 
pretation  in  terms  of  pattern  recognition. 

(3)  Given  a  parsing  of  a  figure,  in  a  great  m  iv  cases,  the  semantic  inter¬ 
pretation  procedure  is  highly  context  depei  lent.  such  a  situation,  a 

primitive  cannot  be  semantically  interpreted  independent  of  the  surround¬ 
ing  primitives  to  which  it  is  syntactically  related. 

(4)  The  elements  of  the  structural  description  anu  the  ease  with  which  the 
structural  description  is  produced  is  dependent  on  the  choice  ol  the 
primitives  and  the  syntactic  relations  between  the  primitives.  However, 
an  important  point  in  which  graphics  differ  Horn  linguistics  is  just  how 
the  parsing  is  to  be  performed.  In  linguistics,  the  primitives  or  lexical 
items  (words)  are  clearly  distinguishable  in  any  terminal  string.  Unfor¬ 
tunately.  in  graphics  this  is  not  always  the  case,  lor  example,  consider 
the  simple  graphic  grammar  composed  of  the  two  primitives,  triangle  and 
rectangle,  and  the  single  syntactic  relation,  on  lop  oj. 
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is  composed  of  a  triangle  on  top  of  a  rectangle  with  their  common  boun¬ 
dary  removed.  But  lor  this  figure  to  be  considered  well-formed,  it  must 
be  parsed  into  a  triangle  and  a  rectangle,  both  of  which  are  implicitly 
contained  within  the  figure.  In  linguistics  the  lexical  items  are  con¬ 
tained  explicitly. 

(.''1  Not  only  are  the  primitives  often  implicit  in  the  pictorial  data  but  it  is 
also  the  case  that  the  syntactic  relations  are  always  implicit  in  the 
data.  Lexical  items  are  concerned  with  only  one  dimension  and  hence 
are  syntactically  related  in  a  string  by  the  singular  relation  of  linear 
juxtaposition,  or  the  relation  of  next  to.  Two-dimensional  pictures  are 
concerned  with  juxtaposition  in  an  infinite  number  of  directions.  The 
significance  of  the  additional  dimensions  to  juxtaposition  can  be  seen 
by  the  example. 


A 

B 

where  A  can  be  considered  on  top  of  B.  but  must  be  soecified  also  to  the 
ntthl  of  or  to  the  left  of  B  to  completely  determine  the  scene. 

(b)  As  indicated  in  paragraph  number  (4).  the  fact  that  the  primitives  are 
contained  implicitly  in  the  pictorial  data  is  different  front  the  occurrence 
of  words  in  a  natural  language.  However,  this  difference  could  be  elimin¬ 
ated  if  the  equivalent  of  words  in  linguistics  are  lines  in  figures.  While 
this  equivalence  will  be  seen  to  shortly  fall  apart,  the  analogy  is  carried 
further.  Both  words  and  lines  are  clearly  defined  in  their  respective 
data.  Higher  level  categories  such  as  NP.  VP,  etc.  are  contained  impli¬ 
citly  in  the  linguistic  data  just  as  higher  level  categories  such  as  triangle, 
square,  etc.,  are  contained  implicitly  in  the  pictorial  data.  Unfortunately, 
the  analogy  collapses  when  the  semantic  aspect  is  considered.  For  while 
words  have  grammatical  and  semantic  significance  in  themselves,  the 
line  has  only  grammatical  significance  in  isolation  from  its  contextual 
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surroutuling.  The  same  difficulty  arises  if  words  are  equaled  to  higher 
level  constituents  such  as  triangles,  for  the  triangle  has  no  semantic 
significance  independent  of  its  use  in  a  specific  figure. 

An  alternative  is  to  consider  letters  of  words  to  be  equivalent  to 
lines  of  a  figure.  In  this  case,  both  letters  and  lines  arc  dearly  defined 
in  their  respective  data  and  neither  have  any  meaning  in  themselves. 

It  may  even  be  argued  that  except  for  figures  composed  of  one  dimensional 
primitives  (bubble  tracks,  etc.)  lines  take  on  no  individual  meaning  even 
within  their  contextual  environment.  This  is  certainly  the  case  for 
letters,  therefore  letters  and  lines  appear  to  be  reasonably  equivalent. 

But  the  analogy  again  can  go  no  further,  for  while  letters  form  words  and 
lines  form  geometric  symbols  it  has  been  seen  that  words  and  geometric 
symbols  tire  not  equivalent  in  their  respective  languages.  However,  the 
equivalence  between  constituents  of  linguistic  data  and  pictorial  data 
can  possib'y  be  made  on  the  following  basis.  Pictorial  figures  can  be 
equated  to  words.  The  primitives  which  comprise  the  figures  tire  equi¬ 
valent  to  the  letters  which  comprise  the  words.  The  lines  which  comnrisc 
the  primitives  are  the  same  as  the  strokes  which  comprise  the  letters, 
f  inally,  the  figures  combine  to  form  a  scene  as  do  the  words  combine  to 
form  a  sentence.  This  equivalence  is  summarized  in  Table  1. 

(7)  A  major  differ encc  in  the  components  which  perform  the  analyses  of  the 
graphic  and  linguistic  data  is  seen  by  the  (act  that  the  syntactic  and 
semantic  components  of  pictorial  analyzers  are  composed  of  essentially 
phrase  structure  grammars.  The  transformational  movement  in  linguistics 
considers  the  semantic  component  to  be  composed  of  what  I'odor  and 
Katz9-10  call  projection  rules.  However,  as  pictorial  analysis  develops, 
any  form  of  the  PSCi  may  very  likclv  be  found  inadequate  for  semantic 
interpretation.  It  should  also  be  pointed  out  that  I'odor  and  Katz  have 
not  been  having  overwhelming  success  with  their  projection  rules.'’ 

The  above  problems  will  be  discussed  in  greater  detail  in  the  following  sections. 
However,  the  fact  that  the  general  syntactic-semantic  approach  is  readily  applicable  to  both 
linguistics  and  graphics  leads  one  to  believe  that,  within  human  conceptualization,  pictorial 
and  linguistic  analyses  are  highly  intermeshed.  One  possible  major  link  between  the  languages 
is  the  Gestalt  factor,  which  to  a  large  extent  has  been  overlooked  in  linguistic  analysis.  The 
Gestalt  factor  will  be  considered  again  in  a  later  section,  though  it  may  be  argued  that  tu  a 
discussion  on  the  competence  model,  ‘.he  Gestalt  factor  is  out  of  place  being  an  aspect  of  the 
performance  model. 

2.6  THE  SYNTACTIC  AND  SEMANTIC  COMPONENT  GRAMMARS 

The  actual  choice  of  a  grammar  in  terms  of  the  primitives  and  a  set  of  relations  which 
are  able  to  provide  a  syntactic  and  semantic  interpretation  to  the  graphic  data  is  a  very  difficult 
one.  Assuming  that  a  number  of  grammars  arc  available,  the  choice  as  to  which  grammar  will 
most  efficiently  and  effectively  process  the  data  is  more  difficult  that  the  similar  problem  in 
linguistics.  If  a  hypothetical  multilingual  machine  is  presented  with  linguistic  data  it  would 
first  search  the  lexical  units  of  the  data  to  determine  with  which  language  it  is  to  be  dealing 
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1  he  decision  as  to  what  language  immediately  limits  the  choice  of  grammars  from  which  the 
machine  can  choose  to  further  analyze  the  data.  As  indicated  earlier,  the  primitive  units  in 
pictorial  input  are  not  so  easily  discernible. 

However,  this  does  indicate  an  approach  to  the  problem.  If  a  graphic  processor  (human 
or  machine)  is  confronted  with  da' t,  it  will  choose  a  grammar  which  seems  most  applicable. 
If  this  grammar  does  not  allow  a  satisfactory  semantic  interpretation  to  be  made,  the  grammar 
if,  discarded,  a  new  grammar  is  chosen  and  the  process  is  repeated.  While  it  may  be  argued 
that  the  choice  among  grammars  is  never  really  made  since  an  individual  has  only  one  large 
multi-leveled  pictorial  grammar,  the  argument  proves  nothing.  Combining  several  small  grammars 
into  one  large  grammar  still  requires  the  parsing  of  the  figures  into  particular  primitives.  Which 
primitives  this  will  be  requires  a  somewhat  tri al-and-error  process  to  take  place.  The  important 
factor  is  the  development  of  a  set  or  orientation.  For  example,  suppose  an  individual  is  pro¬ 
cessing  some  data  and  finds  the  parsing  to  be  reasonably  easy  but  the  semantic  interpretation 
to  be  somewhat  confusing.  When  told  that  what  he  has  been  calling  photographs  of  houses  and 
trees  are  really  photographs  of  paramecium,  he  realizes  his  set  orientation  was  wrong. 


3.  A  SURVEY  OF  THE  LITERATURE 


The  following  brieflv  outlines  some  of  the  syntax-directed  models  which  have  been 
described  in  the  literature.  An  extensive  literature  survey  on  syntax-directed  models  developed 
before  lLHi(i  has  been  made  available  by  Jerome  Feder.'2 

I  he  work  of  It.l.,  (irimsdale12  ct.  at.  was  one  of  the  first  pattern  recognition  programs  to 
use  a  structural  description  of  the  figure  to  be  recognized.  The  recognition  is  performed 
In  comparing  a  statement  describing  the  basic  features  of  the  pattern  to  be  recognized 
to  a  set  of  statements  stored  in  the  computer  which  relates  to  named  patterns.  1  he  statements 
describing  the  figure  ts  found  by  a  scanning  process  which  segments  tne  ligure  into  ,t> roups, 
and  an  assemble  process  which  obtains  descriptions  of  the  groups  determines  the  relation¬ 
ships  between  the  groups,  and  which  compresses  and  codes  this  information  to  lorm  the  state¬ 
ment.  Two  implemented  systems  have  been  produced,  one  using  a  key  word  lor  the  pattern 
to  provide  a  more  rapid  selection  of  the  standard  pattern  statement, 

\t  the  National  Bureau  of  Standards.  Russell  kirsch14  devised  one  of  the  earlier  programs 
which  uses  an  immediate  constituent  grammar  to  analyze  pictures  of  black  and  white  triangles, 
squares,  or  circles.  A  limited  model  accepts  both  pictures  and  sentences  describing  the  pic¬ 
tures  as  input.  1  he  sentences  and  pictures  are  parsed  by  phrase  structure  grammars  and  then 
translated  into  an  intermediate  logical  language.  I  he  logical  interpretation  of  the  sentences 
are  then  tested  to  determine  their  truth  value. 

R.  N arasimhan  1 5 • 1 6 * 1 7 - 1  8 •  ’ 9  while  at  the  l  diversity  of  Illinois,  had  considerable  success 
with  the  syntax-directed  analysis  ot  bubble  chamber  pictures,  A  program  called  UUBBLl. 
SCAN  performs  the  syntactic  analysis.  After  forming  line  elements  from  points  by  a  rectangular- 
array  representation,  the  line  elements  are  labeled  according  to  their  horizontal,  vertical,  or 
diagonal  direction.  lhe  vertices  connecting  the  line  elements  are  labeled  providing  a  labeled 
graph  of  the  picture,  The  line  elements  form  higher  level  constituents  called  tracks  and  the 
tracks  form  the  highest  level  constituents  on  the  basis  of  the  type  of  constraining  vertices. 
Ihc  entire  parsing  process  uses  a  bottom  ujr  scheme.  A  second  program ,  BUBBhlt.  1  A1  K. 
allows  on-line  conversation  in  connection  with  the  analysis  ol  the  bubble  chamber  picture  s, 
It  is  able  ti'  function  as  a  complex  information  retrieval  system  in  order  to  locate  picture 
objects  with  specified  attributes,  Narusimhan  litis  also  dealt  with  the  problems  ot  noise, 
preprocessing,  and  brieflv  with  the  relation  of  syntactic  description  of  pictures  to  the  (iestaft 
phenomena  of  visual  perception,  though  this  last  work  appears  rather  inconclusive. 

While  the  work  on  pattern  recognition  bv  Herbert  F  re  e  man 2  2  1 ,2  2 ,2  3  at  New  \  ork 

1  niversity  does  not  explicitly  use  syntax-directed  analysis,  it  is  interesting  to  note  that  the 
encoding  process  which  he  applies  to  describe  an  object's  contour  is  the  use  ol  a  rectangular 
grid  as  does  Nurasimhan,  Freeman  has  described  manipulations  of  the  codings  such  as  expan¬ 
sion  and  rotation.  For  the  work  on  SAB.  the  direction  grid  has  been  formalized  as  a  grammar 
for  encoding  a  set  of  boundary  points  into  labeled  lines. 

Some  work  on  the  grammatical  formalization  of  handwriting  has  been  done  by  Murray 
1. den  24,28  at  Mil.  1  hough  originally  developed  for  the  generation  of  hand-writing,  the  research 
has  developed  to  the  stage  of  being  used  for  computer  recognition  programs.  Four  basic- 
strokes,  called  segments,  are  defined.  Ihcse  segments  are  transformed  by  rotation,  reflection 
and  translation  or  combined  to  form  IK  strokes  whose  sequences  tire  sulticient  to  describe  the 
English  upper  and  lower  case  letters.  I  lie  rules  for  collating  the  strokes  may  deal  with 
letters  or  with  strokes  between  letters, 
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,H°r  hls  master's  thesis  at  the  University  of  Illinois.  Kenneth  Breeding26  developed  a 
grammar  to  describe  simple  planar  pictures  by  labeled  line  segments  and  selected  vertices. 
I  he  drawings  are  composed  ol  straight  lines  oriented  in  either  a  horizontal  or  vertical  direction. 
1  he  selected  vertices  are  those  which  have  a  degree  of  three.  The  manipulations  of  the  strings 
describing  the  pictures  is  similar  to.  though  surpassed  by,  the  work  of  Freeman. 

Robert  S  Ledl ey 27 ■ 28- 29 - 30  has  done  some  interesting  work  in  the  syntax-directed 
analyses  of  pictures  of  chromosomes  at  the  National  Biomedical  Research  Foundation.  After 
obtaining  the  boundary  ol  an  object  in  a  photograph,  the  object’s  contour  is  analyzed  by  a 
grammar  which  has  live  basic  curves  as  its  terminal  symbols.  These  curves  form  higher  level 
constituents  by  a  bottom-up  parsing  which  determine  whether  or  not  the  object  is  a  chromosome. 
L. cd ley  s  earlier  work  in  the  syntactic  processing  of  pictures  was  concerned  with  what  he  called 
concept  recognition.  One  technique  of  concept  recognition  was  termed  deductive  reference, 
which  was  essentially  pattern  recognition  by  a  bottom-up  parsing  using  a  grammar  for  pictures 
such  as  cartoon-like  houses.  The  second  technique  of  concept  recognition,  inductive  inference, 
uses  a  General  Problem  Solver  approach  by  guessing  a  final  goal  or  recognition  solution  of 
the  picture  and  then  applies  a  set  of  heuristics  to  reduce  the  difference  between  a  syntactic 
description  of  the  guessed  solution  and  a  syntactic  description  of  the  input  picture  to  be  recog¬ 
nized,  Ledley's  use  of  the  GPS  approach  for  an  initial  guess  performs  somewhat  the  same 
process  as  the  abstracting  technique  SAP  uses  in  the  semantic  component. 

William  E.  Miller.  Alan  C  .  Shaw  3  1.32.33  mld  other  nlcmbcrs  of  tho  Computer  Science 
Department  at  Siantord  University  and  the  Stanford  Linear  Accelerator  Center  are  developing 
a  system  which  is  called  a  picture  calculus  to  recognize  and  generate  pictures.  The  picture 
calculus  includes  a  picture  description  language  (PI)L).  rules  for  manipulating  pictures,  and 


processors  for  the  parsing  and  recognition  of  pictures. 

I  he  Primitives  ol  PDF  are  essentially  directed  line  segments  which  have  a  head  and  a 
tail.  The  binary  operators  ol  PDF  describe  various  possible  concatenations  of  the  primitives 
such  as  head  to  tail,  tail  to  tail.  etc.  The  unary  operators  of  PDL,  lor  example,  reverse  the 
tail  and  head  of  a  primitive.  The  language  has  been  used  so  far  for  the  recognition  of  particle 
physics  Pictures  and  other  graph-like  structures  and  to  permit  the  drawing  and  transformation 
ol  line  drawings  on  a  CRT.  The  picture  recognition  scheme,  as  used  on  spark  chamber  photo¬ 
graphs.  applies  a  top-down  parsing  analysis  to  a  string  of  primitives  which  represent  the  iruc- 
t ure  ol  ,he  Picture.  I  he  picture  generation  scheme  also  stores  the  picture  description  as  a 
string,  which  can  be  parsed  to  allow  changes  to  be  made  in  the  picture  description.  Further 
extensions  are  to  include  a  continuous  transformational  operator  which  could  give  the  effect 
of  motion,  the  development  of  PDF  to  utilize  what  might  be  considered  higher  level  topological 
concepts,  such  as  contained  within,  adjacent,  and  above,  and  the  consolidation  of  the  recog¬ 
nition  and  generation  schemes  to  facilitate  learning  by  the  system. 


A  somewhat  different  syntactic  approach  has  been  taken  by  William  Martin34  and  Robert 
Anderson. 35, 34  Martin,  at  MIT,  is  concerned  with  taking  a  mathematical  expression  which  is 
stored  as  a  tree  structure  and  creating  a  visual  display  of  the  expression.  Each  symbol  is 
expressed  in  a  grammar  by  a  special  form,  each  form  is  then  inscribed  by  a  dimensioned  rec¬ 
tangle.  I  he  dimensioned  rectangles  combine  to  create  a  higher  level  dimensioned  rectangle 
which  is  then  centered  on  the  cathode-ray  tube  and  the  contained  expression  displayed.  Ander¬ 
son.  lor  his  I’ll. I).  dissertation,  has  provided  a  grammar  for  the  recognition  of  various  mathe¬ 
matical  expressions  which  are  displayed  in  two-dimensions.  The  model  partitions  a  displayed 
configuration  into  syntact  c  subcategories  which  are  inscribed  in  dimensioned  rectangles, 


The  rectangles  are,  in  turn,  positionally  related,  However,  as  Anderson3^  points  out,  he  is  not 
working  from  a  tree  structure  to  a  display  as  Martin,  but.  from  a  display  to  a  tree  structure. 

M.B.  Clowes,37, 38  at  CSIRO  in  Australia,  has  developed  an  inteiesting  grammar  for 
describing  numerals  in  terms  of  the  contiguous  edges  which  form  their  boundary.  Clowes  has 
also  re.ated  this  work  to  similar  research  in  physiology  and  psychology. 

1  he se  models  all  use  syntax-directed  analysis  for  at  least  some  aspect  of  their  total 
analysis.  A  comparison  of  the  various  models  is  made  difficult,  because  one  uncontrolled 
factor  which,  unfortunately  plays  a  large  role  in  determining  the  overall  models’  appearance 
is  the  type  oi  input  patterns  being  considered.  As  pointed  out  in  the  previous  chapter,  the 
nature  of  the  data  determines  the  type  of  primitives  and  syntactic  relations  which  comprise 
the  data. 

For  example,  the  fact  that  Narasimhan  considered  figures  composed  of  only  one-dimension¬ 
al  constituents  (straight  lines)  eliminated  an  interesting  semantic  problem.  When  a  line  was 
found  in  Narasimhan’ s  bubble  chamber  picture,  it  was  considered  a  track.  Hence,  the  line  is 
able  to  receive  at  least  a  partial  semantic  interpretation  immediately.  However,  in  considering 
the  primitives  used  by  SAP,  they  generally  cannot  receive  a  semantic  interpretation  independent 
of  their  context. 

A  further  simplification  in  analyzing  figures  of  bubble  chamber  tracks  is  that  the  syn¬ 
tactic  relation  between  them  is  singularly  concatenation.  This  allows  the  syntactic  descrip¬ 
tion  to  be  directly  represented  as  a  one-dimensional  string  and  processed  by  a  grammar  for 
semantic  interpretation.  However,  if  the  primitives  are  two-dimensional,  as  in  the  case  of 
SAP,  a  syntactic  analysis  must  first  process  the  figure  to  obtain  some  form  of  explicit  repre¬ 
sentation  of  the  topological  relations  between  the  primitives.  This  representation  may  then 
be  operated  upon  the  grammar  to  receive  a  semantic  interpretation,  However,  it  should  be 
noted  that  both  Narasimhan’s  and  Miller’s  work  has  proved  very  satisfactory  for  the  pattern 
recognition  of  bubble  chamber  tracks.  Similarly,  Ledley  s  pattern  recognition  of  chromosomes 
considers  only  the  syntactic  relation  of  concatenation.  In  all  cases  the  concept  of  continuity 
is  the  determining  factor.  The  difference  in  approaches  essentially  determines  the  number  of 
different  values  under  which  contiguity  is  considered,  It  is  interesting  that  Clowes  does  not 
consider  numerals  to  be  composed  of  one-dimensional  line  segments  but  by  using  a  grammar 
which  describes  a  figure  by  the  line  segments  which  form  the  figure’s  contour,  he  uses  only 
the  relationship  of  next  to. 

The  use  of  figures  whose  primitives  are  one-dimensional  lines  which  require  only  the 
syntactic  relation  of  concatenation  does  not  necessarily  eliminate  the  problem  of  multiple 
parsings  of  the  figure.  As  is  shown  in  Chapter  5,  a  two-dimensional  figure  may  have  a  very 
large  number  of  different  parsings.  This  is  mainly  because  the  topological  relations  which 
have  been  chosen  are  associative  when  serving  as  operators  in  a  syntactic  description.  Ander¬ 
son  avoids  this  problem  by  not  explicitly  using  any  topological  relations  in  the  syntactic 
description. 

While  Grimsdale  uses  topological  relations  between  segments  of  alphanumeric  figures, 
the  semantic  interpretation  is  accomplished  by  phrase  matching  techniques.  The  use  of  a 
grammar  would  be  a  strong  addition  to  his  model.  Though  Cirimsdale's  results  must  be  taken 
cum  grano  satis  (with  a  grain  of  salt),  as  (Jhr39  states,  Grimsdale’s  work  “is  generally  accepted 
as  being  one  of  the  most  powerful  and  -  intuitively  and  psychologically  -  satisfying  of  pattern 
recognition  programs.” 
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Breeding's  consideration  of  figures  composed  of  horizontal  and  vertical  lines  does  no 
more  than  provide  a  notation  for  such  figures.  Its  use  in  pattern  recognition  does  not  appear 
promising.  Kirsch’s  work  is  interesting  and  apparently  has  been  carried  further  in  the  direction 
of  the  syntactic  analysis  of  biological  images.40 

Anderson's  approach  of  syntactically  describing  a  picture  in  terms  of  its  components  and 
pattern  recognizing  the  picture  by  the  positioning  of  coordinates  should  apply  to  most  of  the 
two-dimensional  pictures  which  SAP  considers.  His  method  would  possibly  have  difficulty  with 
some  of  the  structures  because  the  relative  size  of  spacing  between  constituents  is  not  tested. 
The  fact  that  his  syntactic  rules  for  pattern  recognition  are  not  in  the  form  of  a  phrase  structure 
grammar  is  of  no  importance  and  that  they  do  allow  fine  liming  of  the  recognition  process  is 
highly  advantageous,  but  by  not  explicitly  considering  the  syntactic  relations  between  the 
primitives  would  cause  his  model  to  be  cumbersome  if  used  in  a  man  machine  system,  Nara- 
simhan’s  BUBBLE  TRACK  allows  for  a  reasonably  interactive  system.  Miller's  Picture 
Description  Language  is  an  approach  to  this  problem  though  the  lack  of  topological  relations 
is  a  limiting  factor,  Miller32  points  this  out  in  stating  that  the  picture  calculus  "is  not  very 
convenient  for  describing  complex  topological  concepts  without  inclusion  of  concept  recog¬ 
nizers,” 

While  a  phi,  ?  structure  grammar  in  linguistics  can  serve  as  either  a  generator  or  recog¬ 
nizer  of  sentences,  in  two  dimensions  a  single  syntactic  relation  between  two  constituents 
does  not  completely  determine  their  position,  To  get  by  this  difficulty.  SAP  requires  the 
syntactic  relations  to  have  arguments  which  further  position  the  primitives  which  they  relate, 
This  will  be  described  in  detail  in  the  next  chapter.  The  one-dimensional  figures  of  Narasimhans 
rhi.b-t  Utirlf#  igiir  «ire  to  iliiTicttH;  ii  this  i.garJ.  MtILt  Shaw  also.  Af.f.vif 
to  be  able  to  get  by  such  problems  by  using  a  powerful  and  descriptive  set  of  syntactic  oper¬ 
ators. 

A  source  of  information  which  so  far  has  been  used  only  sparingly  in  linguistics  and 
character  recognition  is  contextual  information  between  figures.  This  will  be  discussed  in 
Chapter  6.  Suffice  it  to  say,  that  contextual  analysis  is  a  significant  source  of  information 
which  has  not  been  considered  to  any  extent  by  pattern  recognition  models.  Ledley  uses 
4'srijnir  Tii'fwiiF  ■  m  mr  M-M r  iu  fri  iffy* n;  WJr*  •n-nimrint 

syntax-directed  models  do  not  seem  to  use  any  facet  of  contextual  information.  In  some  models 
this  may  be  due  to  the  fact  that  pictures  such  as  bubble  chamber  tracks  and  mathematical 
expiessions  do  not  contain  a  high  degree  of  contextual  information. 

In  summary,  the  work  of  Narasimhan,  Ledley.  and  Miller  is  similar  to  SAP  in  the  use  of 
a  grammar  to  obtain  a  syntactic  description  of  the  figure.  The  notion  of  contiguity  plays  an 
important  role  in  defining  the  syntactic  relations,  though  SAP  provides  additional  processing 
to  obtain  higher  level  syntactic  relations  while  for  the  most  part  they  need  only  be  concerned 
with  the  singular  relation  of  linear  juxtaposition.  In  addition,  their  use  of  a  grammar  to  obtain  a 
semantic  interpretation  of  the  figure  differs  from  the  manner  in  which  SAP  uses  a  grammar, 
This  is  because  of  the  constraints  which  exist  between  the  constituents  of  the  figures.  The 
phrase  structure  grammar  for  Narasimhan’s  bubble  chamber  tracks  and  Ledley's  chromosomes 
are  context  free  as  is  Narasimhan’ s ' 9  grammar  for  the  pattern  recognition  of  alphabetic  char¬ 
acters.  In  Chapter  6  is  a  description  of  how  the  constraints  which  can  exist  between  either 
constituents  of  a  figure  or  between  figures  of  a  scene  may  be  taken  into  account  by  a  modified 
context  sensitive  phrase  structure  grammar. 

The  next  chapter  describes  a  language  L*  which  can  serve  as  a  meta  language  for  des¬ 
cribing  a  two-dimensional  language  L.  such  as  the  figures  illustrated  in  the  introduction, 
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4.  A  SYNTAX  ORIENTED  LANGUAGE  L* 


4,  t  THE  LANGUAGE 


Ik-  s.l  of  piclu.cs  which  serve  as  inner  to  SAP  are  considered  „>  comprise  a  rvvo- 

,  dPgaaee  I..  As  indicated  in  I'lturc  I.  lire  syntactic  componrnt  or  SAP  Iranslarcs 

'  *  "T*"*"”""'  *‘m'  '  '»  »  »•'  .’"s-din,e„»i„„„l  srrings  of  symbols. 

the’  set  J  L*'r  lhC  SC1  "r  i,n  slfi,llls  *Tiit:li  rcslll[  [t™  Ihe  translation  of 

V  sc  ,»o-d„„ens,o„„l  pretrires  defines  a  language  L*.  Hence,  rhe  syntactic  con, pone, „ 

Pc-fr™m«  a  transit . .  of  a  .»o-di,„ensio„t,l  language  ,o  a  oncdimensional  language 

In  „»ns  orniing  a  r.o.ditnensional  langnage  ,o  a  one-dimensional  language  eerrain  open 
. .  “*■  '»  >«.  a  one -d, mens ional  pattern  completely  determine 

. — t*  i;a"",n-  *• . .  >«««..  ...  w*  m 

arrern  mns,  be  made  esplicr,  in  rhe  o,ie-di,„e„sio„al  parrern.  II,,, «  fhe  one-dimensional  siring 

-  t  which  I,,,,,,  lire  translation  of  the  piernre  I.,.  describes  rhe  . . .  synlacie 

1  call,  res  of  I  Because  the  strings  of  symbols  L*  describe  the  synta,  of  rhe  piciures  ofl  |  . 

nii'>  bc  c°ns  tile  red  a  metalanguage.  The  set  of  strines  I  *  i  ,, 

deserintimi  kiii  '’i  n>  tailed  the  parsings  or  s tructuru I 

description  IMJ)  of  the  picture  I  rin,.  crwi  \  i  * 

i  Ulus.  MHI..)  L*  Ihe  set  of  stnngs  I*  is  sent  to  the 

semantic  component  of  SA|>  to  recc.ve  a  semantic  interpretation. 

of  ..  'i'"d  ‘"“r;  P,USC',"'  ""  ’>•*“»  ""  '.»««.«.  '•*■  The  specification  of  ,hc  svirias 

.  ‘,e!Cn',C‘  '-**■  ".e  prodoe, ion  rules  which  describe  an  algorirhm  for 

‘ “  me”b"  '■*  *«  ",C  «'*“»«”'*  *  '  **■  Thus.  L**  is  ,  sy, noetic  me, ..language 
rclatni.  to  i.  ,  .ind  a  meta-metalanguage  relative  to  L, 

As  indicated  in  the  previous  two  paragraphs,  the  se,  of  strings  L*.  representing  the 
structural  desenpUon  of  a  picture  L,  may  be  obtained  from  the  syntactic  component  of  SAP 
I  c  er  an  alternative  ts  for  a  user  to  provide  SAP  with  a  stnng  I.*  This  allows  a  user  to 
urd  to  he  semantic  component  of  SAP  the  structural  description  of  J  picture  without  having 

"  1  P,c*urc*  ln  ,erms  0f  pitU,rc  recoPnition.  this  provides  a  means  of  testing  the  semantic 

component  independent  of  the  syntactic  component.  In  terms  of  picture  generation,  a  user  can 
send  to  the  syntae.tc  component  of  SAP  the  structural  description.  This  allows  the  picture 
to  be  generated  w„hou.  having  the  actual  label  of  the  ptc.ure.  Thus,  the  syntactic  component 
can  be  tested  and  operated  independent  of  the  semantic  component.  Of  course,  the  user  cn 
send  his  string  to  both  components,  generating  a  picture  from  the  descriptive  string  and  obtain- 
ink  u  label  or  identification  lor  the  generated  picture. 

There  is  a  little  doubt  that  the  facility  of  man  machine  interaction  is  an  important  factor 
m  the  development  and  ultimate  use  of  a  mode,  such  as  SAP.  For  this  reason,  the  syntax  of 
wh.ch  recognizes  the  well -formed  strings  which  the  user  might  send  to  SAP  is  presented 
m  tins  chapter  in  .ts  entirety.  The  process  by  which  the  syntactic  component  ofSAPtrans- 
a.es  a  Picture  L.  ,mo  a  string  or  parsing  L*.  will  be  described  in  the  next  chapter 

Formally,  the  language  L*  is  defined  by  a  phrase  structure  grammar  K2*  which  is  repre¬ 
sented  using  the  Backus  — Naur  Form  (BNF).  The  language  L*  is  a  subset  of  the  sc.  of  ,11 
linilc  strings  of  symbols  from  the  alphah  A.  K2*  is  a  6-tuple.  (P.  S.F.N.R.SS).  The  alphabet 
A  is  the  union  ol  the  sets  of  terminal  symbols  P.  S.  and  F. 

The  vocabulary  of  the  language  I.**  is  a  union  of  the  sets  P.S.F.  and  N.  The  se,  P  is 
se,  of  terminal  symbols  which  tepresen,  the  primitive  geometric  symbols  of  the  language  L . 


-21- 


I  he  set  S  is  a  set  ol  terminal  symbols  which  are  the  names  of  the  syntactic  relations  which  can 
occur  between  the  primitive  symbols.  The  set  T  is  a  set  ol  functions  which  are  the  terminal 
symbols  which  use  the  primitives  and  syntactic  relations  as  arguments.  The  set  N  is  a  set  of 
nonterminal  or  category  symbols  of  L**.  The  set  R  is  a  set  of  production  rules  which  deter¬ 
mine  how  the  syntactic  strings  of  L*  are  to  be  formed.  SS  is  the  goal  of  the  language,  a  syntac¬ 
tic  string. 

4.2  THE  PRIMITIVES 

It  was  stated  earlier  that  there  is  a  great  deal  of  freedom  in  choosing  the  set  of  primitives 
ol  the  language.  However,  it  is  obvious  that  some  primitives  will  represent  certain  pictorial 
data  better  than  others.  For  the  present  work,  the  lollowing  set  of  geometric  primitives  have 
been  chosen: 


I  .  (  ll\lc  ((  ) 


2 .  I  •  1 1 1  p  s  e  <  I  ■ ) 


3.  Rectangle  (R> 


4.  Isosceles  triangle 


5.  Right  triangle 


Right  triangle  down 


7.  Left  triangle 


S.  I  .ell  triangle  down 


Thus.  P  1C  t  R,  Tl.  TR.  TRD.  TL.  TLD  I.  The  difference  between  a  right  triangle  and  a 

left  triangle  is  the  location  of  the  right  angle  of  the  triangle. 

The  entire  set  of  primitives  listed  above  are  completely  determined  by  specifying  their 
height,  width,  and  a  reference  point  on  the  primitive.  The  circle  and  ellipse  have  their  center 
as  their  reference  point  while  the  remainder  of  the  primitives  have  their  lower  left  hand  vertex 
as  their  reference  point.  The  reference  point  is  indicated  by  a  large  dot  on  the  primitives 
shown  on  the  previous  page.  In  addition,  each  primitive  of  a  given  type  in  a  syntactic  string 
must  be  numbered  to  distinguish  it  from  the  other  primitives  of  the  same  type  in  the  syntactic 
string. 

The  general  form  of  a  primitive  is  defined  by  the  syntax  on  this  page.  For  example, 

rectangle  number  3  with  a  width  of  4  and  a  height  of  10  would  be  written  as, 

R ( 3 .4.1 0) 

A  circle  should  have  the  same  horizontal  and  vertical  dimension.  However,  it  is  the  horizontal 
dimension  that  is  used  to  determine  the  diameter  of  the  circle. 


name  cii  •  ::  = 
name  el  lip  ^  = 

<name  rect 
<name  isos  >  ::  = 

<name  rt  tri  >  : :  = 
<name  rt  tridown  c>  : :  = 
name  1ft  tri  : :  = 
•  name  1ft  tridown  >  : :  = 
<zero  >  ::  = 
<number  •>  ::  = 
<integer  >  : :  - 
<numtype  :.  = 
<numtype  : :  = 
''horizontal  dimension  >  : 
•'vertical  dimension  >  :: 
'primitive  arguments  :: 

''circle  >  : 
<ellipse  -»  :: 
''rectangle  > 

<isosceles  triangle  >  ::  - 
''right  triangle  >  f.  - 
''right  triangle  down 

•'left  triangle  s  v. 
'left  triangle  down  >  ::  - 


(.' 

L. 

R 

Tl 

TR 

TRD 

TL 

TLD 

0 

I  2  3  ,  4  !  5  |  6  |  7  |  8  |  9 

<number>  !  <zero>  |  'number  >  <integer 

<integer> 

<integer> 

<integer^ 

'integer.' 

numtype^,  :horizontal  dimension  , 
<vertical  dimension^ 

•'name  cir>  (  ^primitive  argument  ') 

'name  e  1 1  ip>  (  <primitive  argument  •) 
name  rect>  ( ''primitive  argument') 

'name  isos>  (  -primitive  argument  ) 

'name  rt  tri  -  (•  primitive  argument  •) 
name  rt  tridown  •(  ■:  primitive  argument  •) 
'name  1ft  tri  >(•  primitive  argument  •) 
name  1ft  tridown  •  ( •  primitive  argument  •) 


A  figure  represented  by  a  syntactic  string  need  have  only  one  primitive  in  (lie  string 
referenced  in  order  to  position  all  primitives  and  thus  the  figure  in  the  field  of  view,  A  refer¬ 
enced  primitive  has  a  somewhat  different  form  than  that  indicated  above,  the  x  and  v  coordinates 
of  the  reference  point  being  arguments  of  the  primitive.  The  referenced  primitive  may  he  defined, 
as, 

■  x coord  integer  - 

ycoord  integer' 

vrefpt  xcoord  ycoord 

'-reference  primitive  argument  zero  ■,■  horizontal  dimension  -, 

•  vertical  dimension  o  refpt' 

And  thus  added  to  the  definition  of  rectangle  .  etc,,  is, 

•  circle  name  cir  (  reference  primitive  argument  ) 

■  ellipse  ■  name  cflip  (  reference  primitive  argument  ) 

rectangle  name  reel  (  reference  primitive  argument  ) 

■  isosceles  triangle  name  isos(-  reference  primitive  argument  •) 

right  triangle  name  rt  tri  (  reference  primitive  argument  ) 

right  triangle  down  name  rt  tridown  (  reference  primitive  argument  ■) 

left  triangle  name  Ift  tri  •(  reference  primitive  argument  •) 

left  triangle  down  name  Ift  tridown  ■(•  reference  primitive  argument  ) 

primitive  rectangle  isosceles  triangle 

right  triangle  right  triangle  down 
left  triangle  left  triangle  down 
t  irclc  ellipse  • 

For  example,  a  reference  circle  of  di ame  er  15,  and  reference  at  (11.47)  would  he  written  as. 

CM).  15,15.11 .47) 

4,3  THE  SYNTACTIC  RELATIONS 

As  in  the  case  of  the  primitives,  the  choice  of  the  syntactic  relations  are  also  somewhat 
arbitrary.  However,  also  as  in  the  case  of  the  primitives,  an  optimal  or  perhaps  even  feasible- 
analysis  of  the  pictorial  input  necessitates  considerable  thought  to  be  given  as  to  the  choice 
of  grammar  to  be  used.  The  choice  of  the  grammar  for  a  graphic  language  depends  to  a  large- 
extent  on  the  syntactic  relations  desired.  While  it  is  possible  that  at  a  later  date  some  formal 
criterion  lor  choosing  a  graphie  grammar  can  be  presented,  the  best  that  can  be  said  for  the 
present  is  that  intuitively,  it  appears  that  the  higher  level  primitives  should  be  chosen  before 
the  syntactic  relations  if  the  two  choices  can  be  made  separately. 

For  use  in  the  present  research  a  number  of  syntactic  relations  were  tried  which  would 
be  meaningful  in  terms  of  the  defined  set  of  geometric  primitives.  The  final  choice  of  relations 
has  been  extremely  successful  while  providing  a  rather  simple  syntax.  The-  relations  turned 
out  to  be  similar  to  those  suggested  by  Lcdley29  in  his  brief  study  of  the  syntactic  repre¬ 
sentation  of  pieture  expressions.  The  syntactic  relations  arc  of  the  form  * R i (...) ,  where  the 
arguments  contained  within  the  parentheses  will  be  discussed  shortly  and  i=l.,..,6.  The  present 
set  of  binary  relations  consists  of: 

(i)  X,Y,  *R  1  (...)  X  is  on  top  of  Y. 
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or  \n 
ator. 


( i i )  X ,  N  .  * R 2  (...)  X  is  under  V . 

(iii)  X  Y,  *R3  (...)  X  is  to  the  right  of  Y. 

( i v )  X . \  ,  *  R4  (...)  X  is  to  the  left  of  Y . 

(v)  X.Y,  *R5  (...)  X  is  eon  mined  within  Y, 

(\i)  X.Y,  *R(i  (...)  X  eontains  Y. 

here  the  primitives.  X  and  Y,  can  be  considered  operands  and  the  relation  *Ri  is  an  oper- 
Thus.  S  S*Rl.*R2,*R3.*R4,*R5,*R6i, 

The  arguments  of  the  relations  *RI,  *R2,  *R3,  and  *R4  contain  the  following  information. 

(1)  l’he  magnitude  or  distance  of  the  syntactic  relation  (MSR).  (e.g.. 

MSR  0  means  the  primitives  are  touching _ ). 

NO'l  H  MSR  is  optional  if  0. 

(2)  The  *Ri  of  the  syntactic  relation  defines  the  position  between 
primitive  X  and  primitive  Y  is  only  one  dimension  (horizontal 
or  vertical).  To  define  the  relative  position  between  X  and  Y  in 
the  other  dimensions  (called  the  secondary  position )  the  edges 
of  the  primitives  are  used  (e.g,,  TOP  top.  ROT  bottom.  LH  left 
side,  RH  right  side,  1 1C  horizontal  center  line.  VC -vertical 
center  line) 

NO  T I  Optional  if  IIC  or  VC. 

The  edges  of  some  of  the  geometric  primitives  in  use  are  defined 
as: 


I  OP 


Thus,  the  edge  of  a  triangle  may  be  a  single  point,  the  vertex  of 
a  triangle. 

(3)  The  magnitude  or  distance  of  the  secondary  position  (MSP)  between 
the  edges  of  the  primitives  and  defined  by  the  arguments  of  the 
syntactic  relation  must  be  determined. 

NOTH:  MSP  is  optional  if  0. 

(4)  The  secondary  position  between  X  and  Y  defined  in  the  arguments 
of  the  syntactic  relation  must  also  be  given  a  direction,  (e  g., 
HP  up.  I) N  down,  LI  left,  RT  right) 

NOTH.  optional  if  (a)  MSP  0.  or,  (b)  if  any  of  the  following 
combinations  of  edges  of  the  primitives  and  the  direction  of  the 
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relative  position  are  used.  (LE.RT).  (RE.LT):  (BOT, UP).  (TOP. 

DN). 

The  syntactic  relations  *R1.  *R2.  *R3.  and  *R4  are  defined  on 
the  following  page. 

Examples  in  using  the  syntactic  relations  are: 

X.Y, *R1(2. RE3LT)  X  is  2  units  above  Y  with  the  right  edge  of  X  3  units  to  the  left  of  the 

right  edge  of  Y. 

X,Y,*R  1  (2  ,RE3)  =  same  as  above  with  options  used. 

X.  Y,*R3(0.  BOT  l  U  P)  X  is  to  the  immediate  right  of  Y  with  the  bottom  edge  X  1  tin  up  from 

the  bottom  edge  of  Y. 

X,Y.*R3(BOT  I )  same  as  above  with  options  used. 


snamc  sr  s 

*R 

smsr 

:  -  .integer •> 

^  vedg 

TOP  BOT  UC 

hedg 

LE  RE  VC 

<.msp 

:  =  -^integer 

<vdir 

UP  DN 

hdir 

=  ET  RT 

vrelpos  ■: 

:  =  <veug  msp  -vV-dir 

shrelpos 

hedg  msp  -  slidir  ■ 

■sargvsr 

msr  '.  .hrelpos  ■ 

.arghsr  N: 

vmsr '.  .vrelpos'- 

..vertical  relation  1 

:  -  -  name  sr  •  1  (  argv  sr  ) 

vertical  rcl  ation  2 

'.name  sr  ■  2  (  argvsr  ) 

horizontal  relation  3 

name  sr  3  (  arghsr  ) 

horizontal  relation  4 

name  s r  -  4  (  arghsr  ) 

vertical  relation 

vertical  relation  1 

vertical  relation  2  ■ 

horizontal  relation  horizontal  relation  I 

■  horizontal  relation  2 

'directional  relation  vertical  relation 

■  horizontal  relation-' 

The  arguments  of  syntactical  relations  *R5  and  *R6  differ  from  the  other  relations  in  that 
an  additional  argument  or  constraint  is  needed  to  completely  specify  the  contained  within 
relationship,*R5  and  *R6  are  defined  as  follows. 

argewsr  -  zero  vrelpos  hrelpos 

zero  -.  hrelpos  ,•  vrelpos 

contained  within  relation  5  name  sr  5(  argewsr  •) 

contained  within  relation  6  name  sr  6  (■  argewsr  -) 

contained  within  relation  contained  within  relation  5 

■  contained  within  relation  6  ■ 

An  example  of  the  use  of  the  *R5  relation  is  X , Y.*R5((),I.E  1 RT .TO P2DN)  which  says  primitive 
X  is  contained  within  primitive  Y  and  the  left  edge  of  X  is  1  unit  to  the  right  of  the  left  edge 
of  Y  and  the  top  of  X  is  2  units  down  from  the  top  of  Y. 
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Thus. 

<syntactic  relation  vertical  relation 

horizontal  relation 
<contained  within  relation 

The  contained  within  relation  is  actually  a  combination  of  a  horizontal  relation  and 
vertical  relation.  Two  arguments  are  required  for  the  contained  within  relation  since  *R5  and 
*R6  convey  no  information  while  in  the  ease  of  *R  i  lor  LI,... ,4  the  i  specifies  a  particular 
dimension.  Thus,  for  the  first  lour  relations,  only  one  directional  argument  is  required.  Though 
it  can  be  replaced  by  a  combination  of  a  horizontal  and  a  vertical  relation,  the  contained  within 
relation  is  available  to  provide  easier  expressabilitv  by  the  user  but  more  so  it  is  needed  to 
maintain  the  normalized  form  of  the  syntactic  string.  This  normalized  form  requires  two  con¬ 
stituents  to  be  related  by  a  single  binary  relation.  To  use  a  horizontal  relation  and  a  vertical 
relation  in  the  place  of  a  single  contained  within  relation  would  not  conform  to  the  normalized 
form.  That  is,  the  normalized  form  would  be  destroyed  if  two  primitives,  where  one  primitive  is 
contained  within  the  other,  would  be  related  by  two  binary  relations  (a  horizontal  relation  and  a 
vertical  relation)  rather  than  by  a  single  binary  relation  (a  contained  within  relation). 

The  following  list  represents  the  properties  of  the  syntactic  relations  defined  in  this 
chapter. 

(1)  The  entire  set  of  relations  i,  irreflexive.  That  is,  a  binary  relation 
*Ri  is  irreflexive  if  there  is  no  X  such  that  (X.X,*Ri)  . 

(2)  The  entire  set  of  relations  is  unsymmetric.  That  is.  a  binary 
relation  *Ri  is  unsymmetric  if.  for  any  pair  of  elements  X  and  Y 
lor  which  (X,Y,*Ri)  it  is  necessarily  the  case  that  —  ( Y ,X ,*R i). 
where--  is  read  as  not. 

(3)  The  entire  set  of  relations  is  transitive.  That  is,  a  binary  relation 

*Ri  is  transitive  if,  for  any  X.Y,  and  Z,  given  (X,Y,*Ri)  and 

(Y,Z,*Ri)  implies  (X,Z,*Ri). 

Because  the  relations  are  irreflexive.  unsymmetric,  and  transitive  they  are  called  proper 
inequality  relations. 

If  the  *Ri  previously  defined  is  considered  as  a  binary  operation  rather  than  a  binary 
relation  the  following  properties  hold. 

(1)  The  entire  set  of  operations  is  noncommututive.  That  is,  for  any 
X  and  Y,  (X,Y,*Ri)  *  (Y,X,*Ri). 

(2)  The  entire  set  of  operations  is  associative.  That  is,  for  any  X, 

Y,  and  Z,  (X,  Y,*Ri,Z,*Ri)  =  (X,Y,Z,*Ri,*Ri). 

(3)  The  entire  set  of  operations  is  closed.  That  is,  for  every  choice 

of  elements  X  and  Y  in  the  set,  (X,Y,*Ri)  is  also  the  set. 

The  general  form  of  the  syntactic  relations  is  *Ri(n ,  ,n2 ,n3) .  The  arguments  of  the  relation  have 
been  defined  by  the  syntax  as: 

(1)  nj  is  an  integer  providing  the  magnitude  of  the  relation. 

(2)  n2  is  string  of  letters  and  an  integer  to  indicate  the  secondary 
positioning . 

(3)  n3  is  also  a  string  of  letters  and  an  integer  to  indicate  secondary 
positioning  for  a  contained  within  relation.  The  argument  is  null 
and  hence  absent  for  the  horizontal  and  vertical  relations. 

The  syntactic  strings  formed  by  the  primitives  and  syntactic  relations  have  a  normalized 
form  of  a  Reverse  Polish  ordering.  This  notation,  invented  by  the  Polish  philosopher  Lukasiewicz 
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in  1921,  has  been  used  extensively  in  compiling  computer  languages  because  it  makes  the 
operators  and  operands  in  a  syntactic  string  available  at  the  precise  moment  they  are  required 
in  the  compilation.  Early  Reverse  Polish  requires  that  the  operator  immediately  follow  its 
operands,  eliminating  the  need  for  constituent  grouping  by  parentheses,  Thus,  a  segment  of 
the  syntactic  string  can  be  defined  as, 

v.ssss::  =  <primitive>  <primitiveN  Csvntactic  rclations 
vsss'>  cprimiti  ve>  <syn tactic  re  1  ationN 
<primitive><sssx  syntactic  relation'' 

<sssN  'vsss'-  syntactic  relation'' 

where  <sss^  forms  higher  level  constituents  of  the  language.  For  example,  considering  the 
syntactic  relations  and  primitives  without  their  arguments,  a  house  could  be  syntactically 
represented  as: 


where  the  constituent  groupings  arc  actually,  (T 1 1  .((R2.R3 ,*R3).R  I  .* R5).*R  1),  It  should 
be  noted  that  while  there  is  more  than  one  possibility  correct  parsing  for  a  single  graphic  input, 
at  this  time  no  one  parsing  appears  to  provide  any  more  or  less  information  than  an\  other 
parsing,  This  is  an  advantage  over  natural  language  analysis,  since  it  docs  not  appear  nec 
essary  to  obtain  all  possible  parsings  of  the  source  language  statement  (graphic  in  this  case) 
to  obtain  the  full  semantic  content  of  the  statement. 

Using  a  linear  scale  of  1  unit  approximately  equal  to  .  some  further  examples  of  the 
use  of  syntactic  strings  to  represent  graphic  input  are: 

1  KIT 

Tl<  1  .(>.  1 2,).R(  1 ,2 ,5.).*R  1  (O.VC'OI.T) 
or  with  the  use  of  options. 


1T(  1 .6. 1 2). R(  1 ,2,5).*R  I  since  the 
vertical  center  lines  of  the  two 
primitives  are  aligned. 

Rt 
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i '.\c  loin 

R4 

R2 

R 1 


TL(!.3,3).TL(2.3,3).*R4(1  .BOTOl  P),TL( 3,3,3 ).*R4(  1 . BOTOL P). R(3  2.6).R(4.2 .6). 

*R4(2 .  BOTOl'  P),R(2.8.4),*R  1  (O.LE  1  RT).*R4(  1 .  BOTOU  P)  .R<  1 .22 ,7),*R  I  (0.LE2RT) 

or  .,'ith  the  use  of  options, 

TL(  1 ,3. 3).TL(2  3,3).*R4(  1 , B0T).TL(3.3.3).*R4<1  .BOT),R<3.2,6),R<4,2.6),*R4(2.BOT). 
R(2.8.4),*R1(LE1).*R4(1.B0T)R(1.227),*R1(LE2> 

In  the  next  chapter  a  procedure  is  presented  for  testing  the  well-formedness  of  a  syntactic- 
string  in  Reverse  Polish  form. 

4,4  THE  SYNTACTIC  FUNCTIONS 

Some  other  relations  of  the  graphic  language  might  best  be  considered  as  unary  and 
binary  functions  which  operate  on  segments  of  the  syntactic  string.  Some  of  these  are. 

HORIZONTAL  OR  VERTICAL  SYMMETRY: 

This  is  unary  function  which  operates  on  a  segment  of  a  syntactic  string  to  generate  its 
corresponding  symmetrical  syntactic  string.  The  horizontal  symmetry  can  be  either  right  or 
left  and  the  vertical  symmetry  can  be  up  or  down.  The  symmetry  functions  may  be  defined  as. 

'hsym  •::=  *11SYM 
<vsym  >::=  *VSYM 

Csytnf  <hsym  hdir  vsyni  •  <vdir 

For  example,  the  building  drawn  below  can  be  represented  by  the  following  syntactic 

string: 


R4 

R5 


R(l,2,2),R(2,4,6),*R5(TOPl), 

R(3,8,4),*R4(BOT).R(4,2,2). 

R(5.4,6).*R5(TOPl),*R4(BOT) 


using  the  horizontal  symmetry  function,  the  syntactic  string  becomes, 

R(1 ,2,2),R(2,4,6).*R5(TOP  l),R(3,4,4),*R4(BOT)  ,*HSYMR. 

To  produce  the  remainder  of  the  syntactic  string  internally,  the  horizontal  symmetry 
function  performs  the  following  transformations  on  the  syntactic  relations  upon  which  it  operates: 


*R  1  transforms  to  *R  1 

TOP 

transforms 

to  TOP 

*R2  - 

— *  *R2 

HOT 

- ► 

BOT 

*R3  - 

— ►  *R4 

LE 

- ► 

RE 

*  R4  - 

— ►  *R3 

RE 

- ► 

EE 

*R5  - 

— ►  *R5 

HC 

- ► 

HC 

*R6  - 

— ►  *R6 

VC 

- ► 

VC 

The  vertical  symmetry  function,  when  operating  upon  a  syntactic  string,  performs  the  following 
tran  sformation  s : 


*R1  transforms  to  *R2 

TOP 

tra  ns  forms 

to  BOT 

*R2  - 

— ►  *RI 

BOT 

- ► 

TOP 

*  R3  - 

— ♦  *R3 

EE 

- ► 

EE 

*R4  - 

— ►  *R4 

RE 

- ► 

RE 

*R5  - 

— ►  *R5 

1IC 

- ► 

HC 

*R6  - 

— ►  *R6 

ve 

- ► 

ve 

REPEAT  SEGMENT: 


This  is  a  unary  function  which  operates  on  a  segment  of  a  syntactic  string  to  generate 
a  replication  of  the  segment.  The  first  numeric  value  (MBR)  indicates  the  magnitude  of  the 
distance  between  the  replications  where  0  is  optional  and  indicates  touching,  As  defined 
earlier  (UDIR)  and  (VD1R)  are  the  direction  in  which  the  replications  are  to  be  taken,  used  in 
this  case  along  the  horizontal  or  vertical  line.  The  second  numeric  value  (NUR)  indicates 
the  number  of  replications  to  be  made. 

The  repeat  function  may  be  defined  as, 


•"TP 

::=  *RP 

mbr 

::  ''integer 

nur 

: :-  ■'integer 

■'rpf  ^ 

•  rp^ 'mbr  ■'"hdir  • -'nur 

'"rp  'mbr  • '  vd ir  ■  ■  nur 

For  example. 


R I 


id  C  □ 


□  □  □ 
□  □  □ 


R2 


K(  I  (2.2).*R  P I  R  I  2,*R  P  I  l)N2,R(2 . 1  4. 1 4),  *  R.s 
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ENLARGE  OR  REDUCE  SEGMENT 

This  is  a  unary  function  which  operates  on  a  segment  of  a  syntactic  string  to  regenerate 
the  syntactic  string  with  its  various  measurements  changed  by  a  factor  indicated. 

This  function  may  be  defined  as. 


<e  1  s  *ELS 

<rds  >::=  *RDS 

elrdf  <els>  <num>  .  <rds^  ^num> 


For  example. 

R(1.1.1).R<2,3.3).*R5.*ELS2 

becomes 


R(1 ,2.2),R(2.6,6),*R5 


DEFINE  SEGMENT: 

This  is  a  binary  function  which  allows  a  dummy  variable,  DEEi  (i=0,...)  to  be  used  in 
place  of  a  segment  of  the  syntactic  string.  The  segment  of  the  syntactic  string  which  DEEi 
represents  is  defined  as  that  portion  of  the  string  which  follows  DEEi  until  the  first  *DEES  is 
found.  The  integer  i  names  the  particular  segment  of  the  syntactic  string  which  is  being 
repre  sented. 

The  define  function  may  be  defined  as, 

-  defs  *DEES 

defi  *DFF  'integer  • 

^defsf  defi  -'...  •''defs 

ROTATE  ABOUT  REFERENCE  POINT: 

This  is  a  unary  function  which  is  used  to  rotate  primitives  clockwise  from  the  vertical 
around  their  reference  point.  The  numeric  value  of  the  function  indicates  the  number  of  degrees 
to  rotate.  The  rotate  function  may  be  defined  as. 

<rote 
<rotef 

F or  e xample , 


*RO  I  F. 

<rotc>  <integer 


Rf  1 ,4,8),*R()TE45 
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R(1.4,4),R(2,4,4),*R4,R(3,4,4).R(4.4,4),*R4,*Rl.R(7.8,8),*R3(BOT) 

R(6,8.6),R(5,8,8),*R4(BOT),*Rl(LE) 

or 

DEF1,R(1,4,4),*RPRT1.*RPDN1.*DEFS.DEF1.*ELS2.*R5(T0P.RE) 


Figure  4.  An  Example  of  a  Complex  Figure  Defined  by  Syntactic  Functions 


A  final  example  of  the  syntactic  functions  in  F-'igure  4  indicates  sonic  of  the  arbitrarily 
complex  figures  which  can  be  created.  While  many  more  functions  can  be  readily  created,  for 
those  functions  presented  in  this  chapter,  the  syntactic  function  can  be  defined  as, 

unary  syntactic  function'-::-  ^syntf'-  j  .rpf'  <elrdf>  -'rotef- 
vbinary  syntactic  function  ■::=  <defsf  • 

Thus,  to  the  definition  of  the  segment  of  the  syntactic  string  must  be  added, 

vsssN  =  vsss'- s.unarv  syntactic-  function'*  DFJ-i  sss  Dhl'S 

The  goal  of  the  language  is  syntactic  strings.  Any  segment  of  a  syntactic  string  is  also  a 
syntactic  string.  Those  segments  of  the  syntactic  string  which  are  able  to  receive  a  semantic 
interpretation  are  considered  meaningful.  Other  syntactic  strings  are  anomalous.  Hut  in  any 
case,  the  syntactic  string,  as  long  as  it  is  well-formed  Reverse  Polish  string,  can  be  displayed 
graphically,  Thus, 

The  syntactic  relations  defined  by  the  previous  syntax  are  actually  redundant.  That  is, 
only  *R1.  *R3  and  *R5  are  really  needed  to  have  the  same  descriptive  capabilities  as  all  six 
relations.  Thus.  SAP  normalizes  the  syntactic  string,  by  converting,  for  example,  the  *R2 
to  *Rl  with  the  necessary  manipulations  of  the  syntactic  string.  Similarly,  the  functions  are 
removed  from  the  string  before  the  string  is  processed,  the  functions  being  replaced  by  the 
corresponding  segment  of  the  syntactic  string. 

The  normalized  syntactic  string  will  also  be  the  result  of  the  syntactic  component  of 
SAP  for  a  given  picture.  The  syntactic  string  is  a  structural  description  of  the  picture  and  once 
obtained,  is  operated  upon  by  the  semantic  component  to  apply  a  semantic  interpretation  to  it. 
If  a  semantic  interpretation  can  be  applied  to  the  structural  description  then  SAP  has  recognized 
the  pictorial  input. 

The  syntactic  string  is  produced  in  SAP  by  its  syntactic  component.  This  component 
and  the  analyses  it  uses  are  described  in  the  next  chapter. 


5  THE  SYNTACTIC  COMPONENT  OF  SAP 


5.1  AN  OUTLINE  OF  THE  SYNTACTIC  COMPONENT 

The  syntactic  component  operates  on  a  two-dimensional  picture  1..  to  provide  it  set  of 
structural  descriptions  1.*  I  lie  structural  descriptions  are  the  same  one  dimensional  strings 
of  symbols  described  in  the  previous  chapter.  Thus,  the  syntactic  component  obtains  a  set  of 
syntactic  strings,  each  represent'  ig  the  topological  structure  of  a  picture  instead  of  a  user 
presenting  SAP  with  a  syntactic  string  representing  the  picture's  structure,  In  other  words, 
the  syntactic  component  of  SAP  is  a  translator  translating  sentences  (pictures)  of  a  twin 
dimensional  language  L  to  a  one-dimensional  language  L.*. 

(iiven  a  picture  L.,  a  structural  description  of  L.  js  a  representation  of  the  primitives 
and  the  syntactic  relations  between  the  primitive.-  which  comprise  I...  This  description  will 
be  obtained  by  the  syntactic  component  first  as  an  intermediate  tree  structure  and  then  as  a 
linear  string,  The  tree  structure  is  actually  a  pedogogieal  device  to  aid  in  illustrating  and 
conceptional  understanding  of  the  syntactic  analysis, 

To  obtain  the  structural  description  the  syntactic  component  is  composed  of  a  lexicon 
and  two  subcomponents,  a  segmenting  subcomponent  and  a  parsing  subcomponent,  The  seg¬ 
menting  subcomponent  forms  lines  from  points,  higher  level  primitives  such  as  rectangles  and 
triangles  from  the  lines,  and  a  set  of  3-tuples  called  'he  triplet  set  from  the  higher  level 
primitives  (to  be  called  .just  primitives  hereafter)  and  the  syntactic  relations  between  them, 
The  parsing  subcomponent  tests  the  set  of  3-tuples  lor  completeness,  partially  orders  the 
triplet  set,  and  then  transiorms  the  triplet  set  first  into  a  tree  structure  and  then  into  a  Reverse- 

Polish  string  to  obtain  the  final  form  of  the  structural  description  of  the  pictorial  data,  The 

lexicon  defines  the  attributes  of  the  primitives  and  the  syntactic  relations  which  can  exist 
between  the  primitives.  A  flow  diagram  of  the  syntactic  component  is  shown  in  Figure  5. 

5.2  TI.E  SEGMENTING  SUBCOMPONENT 

The  segmenting  subcomponent  must  first  determine  and  define  the  lines  vhich  compose 
the  pictorial  data.  The  lines  are  then  used  to  form  higher  level  constituents,  To  obtain  the 
lines  and  higher  level  constituents  a  grammar  Kl*  is  used,  Kl*  is  a  modified  context  sensitive- 
phrase  structure  grammar  which  is  defined  as  the  4-tuple  (TI,N1.R1.SS),  The  setTI  is  the 
set  of  terminal  symbols  of  Kl*.  By  the  nature  of  the  data,  the  terminal  symbols  are  discrete 

points  which  do  not  explicitly  occur  in  the  grammar,  The  set  N1  is  the  set  of  nontermin¬ 

al  symbols.  The  set  R1  is  the  set  of  production  rules  which  form  higher  level  constituents 
from  the  terminal  symbols,  The  goal  of  the  grammar  is  SS.  the  syntactic  string, 

In  addition  to  the  grammar  Kl*,  the  segmenting  subcomponent  uses  transformation  rules 
on  the  line  segments  formed  by  Kl*.  The  transformation  rules  will  form  additional  line  segments 
or  concatenate  line  segments  already  formed  to  allow  the  parsing  of  the  figure  to  be  obtained, 
In  order  to  obtain  the  line  segments  from  the  discrete  points  it  is  necessary  to  first 
label  the  points,  The  points  are  labeled  by  the  directional  axes  as  shown  on  Page  35, 


I - 1 


Figure  5.  SYNTACTIC  COMPONENT 
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To  label  the  points  the  axes  are  moved  from  point  to  point,  where  at  each  point  all  neighboring 
points  are  labeled  according  to  the  axes.  I  I  us,  the  labeled  points  indicate  their  position 
relative  to  their  neighboring  points.  Any  number  of  directional  axes  may  be  used,  differenti¬ 
ating  between  horizontal,  vertieal.  and  various  diagonal  type  lines.  Note  that  a  single  point 
may  receive  multiple  labels. 

Once  the  points  have  been  labeled,  the  line  segments  are  formed  from  the  following 
productions. 


I  H ( X , X )  PH(X)  - 

l-H(y,x)  '-l>H(y)  •  sl.Hli.x)  '  i  I  H(y.z)  l’H(x) 
Ly(x.x)  l’v(x)N 

Lv(y.x)  ■%(>’)  ’  ‘-I- \,(7..x) '  |  ^1. v (y.z)  l’v(x) 

■  I-  d  R  (x.x)  ^  d  R ( x^  ' 

I.  0  R  (y .  x)  I'dr  (y)  I  -  D  R  I z  ■ x )  '  I  s  I  -  0  R  ^  y  ■ z  ^  I’dR 

Ldl(x,x)  Pdl(x) 

l  -  D  L  ( y '  X )  ^  “"^DL 


(x) 

(x) 


For  example,  a  point  x  labeled  liorr/ontal,  PH(x).  forms  a  horizontal  line  l.H(x.x)  whose  end¬ 
points  arc  x.  i.e.,  a  line  composed  of  one  point  is  formed.  To  extend  this  line  additional 
points  with  the  same  directional  label  as  the  line  are  concatenated  to  the  already  existing  line. 
Concatenating  a  horizontal  point  y,  PH(y).  to  the  left  side  of  a  horizontal  line  bounded  by  points 
z  and  x,  LH(z,x).  forms  a  horizontal  line  bounded  by  points  v  and  x.  1.  (y.x). 

When  a  new  line  (called  an  actual  line)  is  started  because  of  a  change  in  direction  in 
the  points,  the  last  point  of  the  line  just  generated  is  called  a  vertex  point.  Thus,  the  vertices 
are  end  points  of  lines  and  indicate  juxtaposition  points  between  lines.  Vertex  points  always 
have  multiple  labels . 

Once  the  lines  of  the  figure  have  been  determined,  the  input  figure  can  be  defined  as  a 
graph  composed  of  a  set  V  lv.|  of  points  in  F2  and  a  set  of  straight  lines  satisfying  the 
following  conditions 
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1'  I  \  or\  line  contains  precisely  two  points  ot  V,  anil  these  agree  with  its  end  points. 

2.  1  he  lines  have  no  common  points,  except  for  points  of  V. 

1  he  \  ertices  v.  and  vk  which  are  the  end  points  ol  a  line  1  are  said  to  be  incident  w  ith  the 

line,  1  his  ma\  be  written  as  1  (/)  (v.  &  vk)  which  is  read  as  I  joins  v,  and  vk.  where  the  symbol 
(v.  &  vk)  denotes  an  unordered  pair  of  elements  of  v. 

1  he  \  ertices  are  placed  in  a  data  base  where  the  coordinates  ot  each  are  stored  as  attri¬ 
butes  of  the  vertex.  This  information  will  be  used  to  determine  the  length  of  the  line.  Another 

attribute  ot  the  vertex  v  which  is  continually  updated  in  the  data  base  is  its  degree  <5(v). 

which  is  the  number  of  lines  incident  with  v,  a  separate  count  being  kept  for  actual  lines. 
Each  line  is  assigned  two  arguments,  the  vertices  which  bound  the  line.  The  arguments  are 
defined  as  follows: 

(i)  l  or  l-H  ( v • , v | )  the  first  argument  is  the  left  hand  vertex  which  bounds 
the  horizontal  line. 

(ii)  bor  Lv  (v.,v.)  the  lirst  argument  is  the  lower  vertex  which  bounds  the 
vertical  line. 

(in)  bor  Ldr  ( v j , v j )  or  Ldl  (v,.Vj)  the  first  argument  is  the  left  hand  vertex 
which  bounds  the  diagonal  line. 

Since  the  semantic  component  requires  the  structural  description  to  be  composed  of  higher 
level  primitives  than  lines,  additional  rules  must  be  added  to  the  grammar.  This  would  not 
necessarily  be  the  case  il  the  pictorial  data  were  one-dimensional  figures,  bor  the  pictorial 
data  being  considered,  the  higher  level  primitives  are  the  triangles  and  rectangles  described  in 
the  previous  chapter,  The  input  figure  will  be  parsed  into  the  higher  level  primitives  by  combining 
the  lines  determined  by  kl*.  bor  example,  the  figure 


can  be  readily  parsed  into  three  rectangles  and  a  triangle, 

However,  a  difficulty  in  parsing  the  pictorial  data  arises  when  the  higher  level  primitives 
required  by  the  semantic  component  are  only  implicitly  contained  in  the  data.  This  occurs  if, 
for  example,  the  primitives  are  two-dimensional  and  the  common  boundary  between  two  primitives 
is  removed.  In  this  case  the  segmenting  subcomponent  must  add  lines  to  the  figure  to  make  the 
primitives  explicit.  Because  there  may  be  a  choice  as  to  which  lines  to  add  to  the  figure,  more 
than  one  configuration  of  the  figure  may  be  possible. 

For  example,  assuming  that  the  structural  description 
is  to  have  rectangles  as  its  primitives,  consider  the  .  ^ 

figure  on  the  right:  I  I 


There  are  lour  different  configurations  for  the  figure,  lhey  are. 


•  •  •  # 

• 

• 

• 

• 

•  •  •  • 

where 

the 

dotted 

lines  are  the 

added  line 

s.  To 

produce  these  configurations  the  following 

transformation  rules  are  used.  These  rules  are  not  considered  part  ol  the  grammar  Kl*  because 
of  the  nature  of  their  operation.  The  brackets  on  the  right  hand  side  indicate  a  choice  m  the 
lines  to  be  added  and  the  numbers  on  the  right  hand  side  of  the  transformation  rule  indicates 
that  the  corresponding  elements  on  the  left  hand  side  ol  the  rule  are  carried  over. 


l  H  <  v!  vi>-  1  V(vi  vk>  - ► 


L  H  ( v  j .  v  i  M  v  * v  j  ■ v  k  ^ 


(  (1)  .(2)  •  ) 

\  (1)  .  (2)  .  I-  v  <  vm  .  Vj)  ( 

j  (1)  •  (2)  .  L  H<vm.v,)  i 

}  (1)  ,  (2)  .  1-  V(vm.v.)  j 


The  added  lines  (called  artificial  lines)  are  designated  by  a  prime  so  that  the  parsing  sub¬ 
component  will  be  able  to  indicate  in  the  structural  description  that  the  lines  were  added  by  the 
segmenting  subcomponent. 

Note  that  when  artificial  lines  are  added  new  vertices  may  be  created.  The  Appendix 
10.2  contains  the  entire  set  of  rules  necessary  to  add  artificial  lines  to  the  pictorial  data  being 
considered.  Each  use  of  an  operation  rule  adds  one  artificial  line  to  the  figure.  All  lour 
configurations  of  the  sample  figure  would  be  operated  on  by  the  parsing  subcomponent,  producing 
structural  descriptions  (parsings)  to  be  sent  to  the  semantic  component.  It  is  possible  that 
all  of  the  parsings  will  receive  the  same  semantic  interpretation  or  that  some  of  the  parsings 
cause  the  figure  to  be  considered  anomalous,  due  to  limitations  in  the  semantic  component, 
and  hence  assigned  no  semantic  interpretation. 

Once  the  adding  of  artificial  lines  to  the  pictorial  data  is  complete,  it  is  necessary  to 
explicitly  define  the  higher  level  primitives  which  comprise  the  data.  I  his  is  done  by  combining 
the  lines  into  the  higher  level  constituents  such  as  triangles  and  rectangles  by  the  following 
steps. 

(1)  Artificial  lines  which  were  added  by  the  segmenting  subcomponent  are 
concatenated  with  actual  lines  of  the  figure.  Two  lines  are  considered 
for  concatenation  if  an  artificial  line  has  the  same  direction  as  an  actual 
line  and  they  have  a  common  end  point. 

(2)  The  artificial  and  actual  lines  are  concatenated  if  their  common  vertex 
v  has  a  degree,  5(v)-3. 

(3)  If  S(v)  4,  the  artificial  and  actual  lines  cannot  be  concatenated. 

If  two  lines  are  concatenated,  the  two  lines  before  the  concatenation  are  not  lost  since  they 
may  also  be  needed  in  the  formation  of  higher  level  constituents.  The  concatenated  line  is 


primed.  This  may  be  written  as  shown  below  where,  for  the  first  rule,  the  new  line  formed 

is  1  'l  < v , .  v k ) . 


l-1(vi,vi).L1(vi.v|t),Lm(v..vn) - ►  L  Vvi-vk>'(3) 

L1(v,.vi).L1(vi.vlt).Lm(vn.vj) - *>  L  ',(v„vk).(3) 

Once  all  possible  line  concatenations  are  made,  a  grammar  will  form  the  higher  level  primitives 
required  by  the  semantic  component.  The  grammar  rules  which  form  these  constituents  are 
of  the  form. 

<X(w.h.v)  ':!=  <  (v, ,  Vj)>  <L  ^  vk ,  V|)>. . .  '0 

The  rule  states  that  constituent  of  type  X  may  be  formed  from  the  juxtaposed  lines  in  the 
context  of  restriction  0.  The  arguments  of  constituent  X  are  defined  as: 

(i)  w  is  a  number  assigned  to  each  constituent  identifying  that  constituent 
from  other  constituents  of  the  same  type; 

(ii)  h  is  the  horizontal  dimension  of  the  constituent: 

(lii)  v  is  the  vertical  dimension  of  the  constituent. 

The  primes  enclosed  in  parentheses  indicate  they  are  optional  and  thus  a  line  may  be 
artificial,  actual,  or  a  concatenation  of  the  two.  The  restriction  W  refers  to  an  entry  in  the 
lexicon  which  contains  additional  restrictions  for  forming  the  constituent  and  defines  the 
attributes  of  the  constituent  being  formed.  The  lexicon  entry  indicates  how  the  horizontal 
and  vertical  dimensions  of  the  constituent  are  to  be  determined  and  other  attributes  which  will 
be  required  to  complete  the  structural  description  of  the  figure.  When  the  constituent  is  formed, 
its  entry  is  made  in  the  data  base  and  the  vertices  which  are  the  arguments  of  the  line  s 
forming  the  constituent  are  listed  as  an  attribute  of  the  constituent. 

Examples  from  the  set  of  production  rules  to  form  the  higher  level  primitive  constituents 
required  for  the  pictorial  data  being  considered  are  shown  in  Figure  6.  The  listing  of  the 
grammar  Kl*  in  Appendix  10.2  contains  the  complete  set  of  rules  for  all  eight  primitives 
indicated  in  Chapter  4. 

In  contrast  to  juxtaposed  elements  in  linguistics,  the  above  rules  do  not  require  the 
juxtaposed  elements  to  be  ordered.  The  arguments  of  the  lines  are  ordered  and  thus  eliminate 
any  possible  ambiguity  between  figures.  For  example,  assume  the  following  rule  where  the 
arguments  are  not  considered  ordered  (TS  stands  for  scalene  triangle) 

•'•TSfw.h.v)  -  ''l'0R(vi'vj)  'LDR(vk-vi);''r,'H(vi'vk) 
so  either  of  the  two  figures  may  be  produced. 


-39- 


<-R(w.h,v)  ^1-H  (v,,V|)N  l  -  v  ( v  s ,  v  ^ )  <LH(V|,.V|) 

<Lv(Vj.V|)^  RECTANGLE 


i 


TI(w,h,v)  L^v.^)  •  LDR<v.,vk)  ■ 

lD L (vk -vj)  ISOSCELES  TRIANGLE 


<C(w,h,v)  < L H ( v . , v j )  -  L D  L  ( v j ,  v k ) ^  ■  T-  v  I v i - v k ^ 

<LOR(vrv>><LH<vn'VJ  ’*  LDL<vp'V") 
<Lv(vp-v,)  <Lo  r (v,. v,) ■'/  CIRCLE 


Figure  6.  Examples  of  Production  Rules  to  Form  Higher  Level  Primitives 
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Hut  it  the  arguments  of  the  lines  are  considered  ordered  and  def  ined  as  earlier,  only  the  left  hand 
figure  is  possible. 

Having  formed  all  the  higher  level  primitives  from  the  actual  and  artificial  lines,  in  order 
to  provide  a  structural  description  of  the  pictorial  data,  a  set  of  3-tuples  are  created,  composed 
of  pairs  of  contiguous  primitives  and  their  respective  syntactic  relation.  The  primitives  are 
paired  bv  finding  a  subset  of  the  vertices  of  one  primitive  to  also  be  a  subset  of  the  vertices 
of  a  second  primitive.  The  correct  syntactic  relation  is  determined  by  their  definitions  in  the 
lexicon. 

I  he  general  form  of  a  lexicon  entry  for  cither  a  primitive  or  a  syntactic  relation  is  of 
the  following  form. 

NAMlv.  form 

Requirements:  F(1  )VU  (i(lt) 

Assignments:  m.  j  k(  I m )  0|\V ( 1 ,) 

K 

*  he  entry  NAMH  can  be  either  the  name  of  a  primitive  or  a  syntactic  relation.  The  portion 
labeled  Requirements  lists  any  restrictions  which  SAP  must  consider.  1  he  functions  F  and  Ci 
represent  general  functions  such  as  length  of  a  line  segment  and  VU,  is  an  arithmetic  relation 
or  a  relation  such  as  contiguous.  "IE  .  The  portion  labeled  Assignments  assigns  to  variable 
mt  ei(her  (i)  a  numeric  value  which  results  from  the  calculation  of  the  indicated  expression 
where  k  and  W  are  functions  which  obtain  the  x  and  v  coordinates  of  the  arguments  Im  and  I, 
which  are  line  segments  or  vertices,  or(ii)  a  label  n^  which  indicates  the  directional  value  of 
an  argument  of  a  syntactic  relation.  The  lexicon  entries  for  the  eight  primitives  and  the  three 
basic  syntactic  relations  are  listed  in  Appendix  9.2.  An  example  of  the  entry  for  the  relation 
oi i  lop  oj  is  shown  below, 

ON  TOP  OF:  X.  Y.*R  l(n,  .n/^i^) 

Requirement  :  X(BOT)  fflY(TOP) 

Assignments  :  nj  (| 

n  2  IF. 

if  XCOORD(XL  E)  XCOORl)(YLE) 
then. 

n3  XCOORD(X[_  E)  -  XCOORD(Yl  e) 

11 4  RT 

if  X C 0 0 R D ( X L  E )  XCOORD(Yle) 
then, 

n3  =  XCOORD(Yle)  -  XCOORIMXL  e) 
n4  LT 

The  3-tuples  constitute  a  set  of  triplets,  T  ,  It,  I.  Thus  triplets  arc  of  the  form,  t. 
(l,h-P|'sm)  where  pk  and  p,  are  members  of  the  set  P  of  primitives  and  sm  is  a  membe  r  of  the 
set  S  of  syntactic  relations,  and  where  p^  and  p,  are  contiguous  primitives,  and  sm  is  the 
syntactic  relation  between  them.  The  definitions  of  the  syntactic  relations  may  be  found  in 
Chapter  4.  While  the  syntactic  language  in  Chapter  4  defined  six  syntactic  relations,  only 
three  of  them  are  actually  necessary, 

*R1  on  top  of 

*R3  to  the  left  of 

*R5  is  contained  within 


-  li¬ 


the  remaining  three  being  just  the  inverse  of  the  preeeding  three.  To  normalize  the  graphie 
data  and  reduce  tire  number  of  different  structural  descriptions  which  must  be  considered, 
only  the  above  three  syntactic  relations  are  used  internally  by  SAP,  II  a  user  provides  SAP 
with  a  syntactic  string  using  relations  other  than  the  three  indicated,  the  relations  are  converted 
to  the  three  above  relations  by  easily  manipulating  the  syntactic  string. 

Triplets  may  also  be  of  the  form  (X.(),0),  This  occurs  if  the  primitive  X  occurs  in  the 
figure  but  is  not  contiguous  to  any  other  primitive  in  the  figure,  'Triplets  of  this  and  other 
forms  require  various  tests  and  transformations  to  be  performed  on  the  triplet  set.  Once  all 
possible  triplets  have  been  formed,  the  triplet  set  T  is  sent  to  the  parsing  subcomponents  where 
it  is  checked  for  completeness  and  then  transformed  into  a  structural  description  of  the  figure, 

5.3  THE  PARSING  SUBCOMPONENT 

5.3.1  COMPLETENESS  TESTS 

Once  the  triplet  set  has  been  formed  from  all  pairs  of  contiguous  primitives  and  the 
syntactic  relations  between  them,  the  triplet  set  is  tested  by  the  parsing  subcomponent  for 
completeness.  To  facilitate  explaining  the  completeness  tests  the  triplet  set  will  be  repre¬ 
sented  by  a  directed  graph.  Each  edge  of  the  graph  has  associated  with  it  an  orientation 
resulting  from  the  end  points  of  each  edge  which  constitute  an  ordered  pair  of  vertices, 

The  directed  graph  D  is  a  triple  (P.S.A)  consisting  of  a  nonempty  set  P,  a  set  S  which 
is  disjoint  from  P,  and  a  mapping  A  of  S  into  P  X  P.  P  X  P  is  the  cartesian  product  of  a  set 
P  with  itself  forming  all  ordered  pairs  (p.,p.)  such  that  pfC  P  and  PjC  P, 

(i)  The  elements  of  P  are  called  vertices  and  represent  the  primitives  which 
are  contained  in  the  triplets, 

(ii)  The  elements  of  S  are  called  directed  edges  and  represent  the  syntactic- 
relations  which  are  contained  in  the  triplets. 

(iii)  A  is  called  the  direct  incidence  mapping  associated  with  D. 

If  s  C  S  and  A(s)  =  (p.,pj)  then  the  directed  edge  is  said  to  have  P|  as  its  initial  vertex 
and  P|  as  its  terminal  vertex  or  s  ==  (p;,Pj). 

Thus,  the  directed  graph  is  isomorphic  to  the  triplet  set.  For  example,  the  triplet  set 
(« , 6 , * R i , ) ,  (/'-3,y,*Ri2),  (/d,<5,*Ri3),  (y.5,*Ri4)  would  form  the  directed  graph, 


■  I 


It  is  interesting  to  note  that  the  directed  graph  representing  the  triplet  set  could  be 
obtained  directly  from  the  two-dimensional  picture  serving  as  input.  As  indicated  in  Chapter 
Two,  the  figure  itself  may  be  considered  a  geometric  graph.  A  modified  dual  of  this  geometric 
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graph  is  the  desired  directed  graph  representing  the  triplet  set.  To  obtain  the  modified  dual 

(i)  Consider  the  geometric  graph  (figure)  with  regions  R .(i  =  1 , n).  Associate 

a  point  p.  with  each  region  R.  by  choosing  one  of  the  points  within  the 
region.  II  two  regions  R;  and  Rj  are  adjacent,  join  p.  and  Pj  by  an  edge 
P i P I  which  intersects  the  common  boundary  of  R|  and  R.  only  once  and 
has  no  point  in  common  with  any  other  boundary  of  the  graph. 

(ii)  No  point  p.  is  assigned  to  the  region  surrounding  the  graph. 

(iii)  The  edge  pjPj  receives  a  direction  and  label  which  corresponds  to  the 
syntactic  relation  ordering  between  regions  R.  and  Rj. 

This  procedure  yields  a  new  graph  D'  with  vertices  p, . Pn.  it  is  called  a  modified  dual 

graph  of  D. 

For  example,  consider  the  picture  below. 


The  point  p,  of  region  R,  is  indicated  by  the  dot  and  labeled  by  the  greek  letter  shown.  Connect¬ 
ing  the  dots  by  the  indicated  dotted  lines  and  giving  a  direction  and  label  to  the  corresponding 
connections  provides  the  following  the  directed  graph. 


a 


n 


While  the  completeness  tests  are  presented  in  terms  of  the  directed  graph  to  provide 

ease  of  explanation,  the  tests  are  also  described  in  terms  of  the  transformation  rules  which 

actually  operate  on  the  triplet  set.  The  general  transformation  rule  is  of  the  form,  T1 - *T2, 

where  T1  and  T2  are  strings  of  triplets  of  the  form  (a  j  ,/3  1  +Rj }  ),(a  ,  ,/3 2*  Rj2) , - . ,(an  ,*Rj  ). 

T1  is  a  subset  of  the  original  triplet  set  T  and  T2  is  the  subset  Tl  after  it  has  been  transformed. 
As  indicated  in  the  previous  section,  triplets  may  also  be  of  the  form  (a, 0,0)  where  ft.  and  *Rj. 
are  zero.  As  a  shorthand  notation,  T2  may  have  members  of  its  string  of  the  form  (n)  which 

indicates  that  the  n,h  triplet  on  the  left  hand  siUe  of  the  rule,  Tl,  should  be  transferred  to  the 
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right  hand  side,  T2,  unmodified.  Triplets  may  be  removed  from,  added  to,  or  modified  in  the 
triplet  set.  The  order  of  the  triplets  in  T1  and  T2  is  arbitrary,  though  it  will  be  shown  in  a 
later  section  that  it  is  necessary  to  partially  order  the  triplet  set.  The  arguments  of  the  primitives 
and  syntactic  relations  which  were  described  in  the  last  chapter  are  not  included  in  the  examples 
of  this  chapter  as  a  matter  of  convenience. 

There  are  three  completeness  tests  which  are  performed  on  the  triplet  set.  These  tests 

are . 

1.  The  test  for  isolated  primitives.  An  isolated  primitive  occurs  when  a 

primitive  is  contained  within  another  primitive  in  the  figure  and  the 

former  primitive  is  not  contiguous  to  any  primitives. 

2.  The  test  for  missing  relations.  A  missing  relation  occurs  when  two 

primitives, which  are  not  contiguous,  are  both  syntactically  related  to 
a  third  primitive  by  the  same  type  ol  relation. 

3.  The  test  for  inconsistent  relations.  An  inconsistent  relation  occurs 

when  two  contiguous  primitives  are  syntactically  related  when  they 
should  not  be. 

The  completeness  tests  will  now  be  described  in  detail. 

1.  The  test  for  isolated  primitives  is  essentially  a  check  to  determine  whether  the  undirected 
graph  Cl  which  corresponds  to  the  directed  graph  1)  is  connected.  Hie  test  for  connectedness 
of  the  undirected  graph  may  be  defined  as  follows: 

A  finite  sequence  Sj,s2 . sn  of  edges  of  the  corresponding  undirected 

graph  constitutes  an  edge  progression  of  length  N  if  there  exists  an 
appropriate  sequence  of  n  <  1  vertices  p0,p]....pn  such  that  Sj  (p(  &p|) 

for  i  =  1,2 . n.  The  set  of  edges,  without  regard  to  sequencing,  is  said 

to  constitute  a  chain.  Finally,  a  graph  is  said  to  be  connected  if  every 
pair  of  distinct  vertices  are  joined  by  at  least  one  chain.  Other  graphs 
are  said  to  be  disconnected.41 

The  graph  G  is  not  connected  if  its  edges  can  be  partitioned  into  two  subsets  S,  and  S2  such 
that  both  end  points  of  every  edge  are  in  the  same  subset.  Ihere  are  two  cases  of  isolated 
primitives  to  he  considered,  depending  on  whether  I5,  and  or  P2.  the  subsets  resulting  from 
the  partitioning  oi  the  set  oi  vertices,  consists  ol  an  isolated  vertex  p.  that  is.  5(pj)  0. 

(i).  The  first  case  of  an  isolated  primitive  arises  when  P,  or  P2  consists  of  a  single 
vertex.  Hence,  either  S,  or  S2  is  null.  This  may  occur  in  a  figure  when  a  primitive  X  is  contained 
within  a  primitive  Z.  but  primitive  X  is  not  contiguous  to  primitive  7  or  any  other  primitives 
which  may  be  within  Z.  For  example. 
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The  directed  graph  tor  this  example  is 

Z 


The  undirected  graph  is. 


W 


*R5  X 


*R5 


X 


To  form  a  connected  graph  the  vertex  X  must  be  connected  by  an  edge,  representing  a  syntactic- 
relation.  to  either  Z  or  W.  Since  in  the  general  case  there  may  be  several  primitives  contained 
within  Z  to  which  X  can  be  syntactically  related,  X  will  be  related  to  Z,  the  terminal  point  of 
the  contained  within  relation.  The  resulting  undirected  and  directed  graphs  are, 

*R5  *R5 

Z  -  X  z - ►  X 


*R5 


and 


*R5 


W  W 

The  triplet  from  which  the  isolated  primitive  results  is  of  the  form  (y,0,0).  1  he  triplet 

set  for  the  example  is  (W,Z.*R5),  (X,0,0)  and  it  is  transformed  to  (W .  Z,  *R5),(X,Z  ,*R5).  The 
transformation  rule  tor  this  type  of  isolated  primitives  is 

(a./3.*R5),  (>  ,0,0) - -  (1) .  (y,/d.*R5) 

The  triplets  of  the  left  are  members  of  the  original  triplet  set.  The  triplets  on  the  right  are  the 
result  of  a  transformation  on  the  original  triplets,  the  (1)  indicating  that  the  first  triplet  remains 
the  same  while  the  second  triplet  is  transformed  as  indicated. 

The  first  case  of  an  isolated  primitive  with  P,  or  P2  consisting  of  a  single  vertex  may 
also  occur  when  a  primitive  X  is  contained  within  a  primitive  Z.  and  while  X  is  contiguous  to 
at  least  one  primitive  Y  which  is  contained  within  Z,  there  are  no  primitives  contiguous  to  Z. 
For  example, 

,  Z 


The  corresponding  directed  and  undirected  graphs  are, 
Z  Z 

and 


*R2  *R2 

To  form  a  connected  graph  the  vertex  Z  is  connected  by  an  edge  to  one  of  the  primitives 
contained  within  Z,  the  choice  being  arbitrary.  The  resulting  undetected  and  directed  graphs 


-45- 


are,  choosing  X  for  instance, 

Z 

*R5  and 


Z 

}  *R5 


X  -  Y 


X  - *  Y 


*R2  *R2 

The  triplet  from  which  the  isolated  primitive  results  is  again  of  the  form  (y  ,0.0).  The  triplet 
set  fo;  the  example  is  (Z,0,0)  (X,Y,*R2)  and  it  is  transformed  to  (X,Z,*R5).(X,Y.*R2).  The 
transformation  rule  for  this  type  of  isolated  primitive  is, 

(a./d,* Ri).  (y .0,0)  - ►  (1),  (a,y,*R5) 

In  the  actual  implementation,  the  choice  as  to  which  of  the  two  transformation  to  use  is 
made  by  checking  the  coordinates  of  the  vertices  of  the  primitives.  The  vertices  are  attributes 
of  the  primitives  though  they  have  not  been  indicated  here. 

lhe  occurrence  where  both  P,  and  Pj  consist  of  a  single  vertex  can  be  easily  handled. 
This  arises  when  primitive  X  is  contained  within  primitive  Z,  and  both  X  and  Z  are  contiguous 
to  no  primitives.  For  example. 


Z 


X 


The  directed  and  undirected  graphs  are, 

Z  X 

which  may  be  connected  to  form, 

Z - *X  and,  Z - X 

*R5  *R5 

The  triplet  set  for  the  example  is  of  the  form  (Z,0,0)  tX.0.0)  and  is  transformed  to 
(X,Z,*R5).  The  transformation  rule  for  this  type  of  isolated  primitive  is, 

(a, 0,0),  (0,0,0) - ►  (/3,n,*R5) 

(ii).  The  second  case  of  an  isolated  primitive  arises  when  the  partitioning  of  the  uncon¬ 
nected  graph  Ci  provides  two  subsets  P1  and  P2,  both  of  which  contain  more  than  one  vertex. 
This  occurs  in  a  figure  when  a  primitive  X  is  contiguous  to  at  least  one  primitive  Y,  and  while 
both  X  and  Y  are  contained  within  a  primitive  Z,  neither  X  nor  Y  is  contiguous  to  Z  or  any 
other  primitives  which  are  contained  within  Z.  For  example, 

X 


1 
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The  directed  is 

Z 

*R  5 
W 


*R2 

X  - *  Y 


to  which  is  added  a  relation  between  Z  and  either  of  the  primitives  not  connected  to  Z. 

The  triplet  set  for  the  example  is  of  the  form  (W.Z.^RS),  (X.Y.*R2)  and  is  transformed  to 
(W.Z.*R5).  (X.Y.*R2)  (X.Z.*R5).  The  transformation  rale  is. 

ia.fi  * R5>,  (y.8.*Ri) - ►  (I).  (2).  iy.fi  * R5) 

As  with  the  previous  transformation  rules,  their  application  is  in  part  determined  by  checking 
the  attributes  of  the  primitives. 

2.  The  lest  for  missing  relations  is  to  determine  when  two  noncontiguous  primitives  should  be 
related.  The  addition  of  these  relations  allows  further  parsings  of  the  figure  to  be  performed, 
the  adding  of  relations  being  the  inclusion  of  more  triplets  in  the  triplet  set. 

An  example  of  a  figure  which  results  in  a  missing  relation  is. 


The  triplet  set  obtainable  from  the  contiguous  primitives  is. 

( X ,  Y .  *  R 1 ) .  ( X ,  Z .  *  R I ) 

The  directed  graph  formed  from  these  triplets  is. 


*R  1 


Y  Z 


*  R I 


It  will  be  shown  in  the  next  section  that  two  parsings  may  be  obtained  from  the  triplet  set. 
A  third  relation  can  be  represented  between  Y  and  Z.  The  triplet  representing  this  relation  is 
(Y,Z.*R2)  and  by  the  addition  of  this  triplet  to  the  triplet  set  four  more  parsings  of  the  figure 
can  be  obtained.  After  the  third  relation  is  added  the  directed  graph  is. 

X 


The  missing  relation  test  searches  the  directed  graph  for  the  presence  of  two  or  more  directed 
edges  which  have  the  same  syntactic  label  and  the  same  initial  or  terminal  vertex.  A  relation 
is  considered  missing  if  the  noncommon  vertices  of  the  directed  edges  just  described  are  not 
connected  by  a  path  which  does  not  contain  the  common  vertex.  There  are  two  types  of  cases 
in  which  a  missing  relation  may  arise,  depending  on  whether  the  common  vertex  is  an  initial 
or  terminal  vertex. 


I 
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If  the  directed  edges  with  the  same  syntactic  label  have  the 


same  initial  vertex. 


the  missing  relation  is  between  fi  and  y  where  *Ri  cannot  be  a  contained  within  relation.  It 
will  be  pointed  out  in  the  discussion  on  the  inconsistency  test  that  *Rj  also  cannot  be  a  con 
tained  within  relation,  and  of  course  i  -  j.  The  transformation  rule  which  operates  on  the  triplet 
set  for  the  situation  of  the  directed  edges  having  the  same  initial  vertex  is, 

(u./3,*Ri),  («,y,*Ri) - *(l),  (2),  (£,y,*Rj) 

The  directed  graph  for  the  directed  edges  having  the  same  terminal  node  is, 

«  V 


where  again  there  is  a  missing  relation  between  a  and  fi. 
missing  relation  is, 


An  example  of  a  figure  with  such  a 


which  has  the  directed  graph, 


*R  1 


*R  1 


As  pointed  out  earlier,  by  adding  a  triplet  which  relates  X  and  Y  to  the  triplet  set  will  allow 
additional  parsings  of  the  figure  if  *R1  is  not  a  contained  within  relation. 

However,  if  *R1  is  a  contained  within  relation  it  is  mandatory  to  determine  the  missing 
relation  between  X  and  Y.  Because  primitives  related  to  Z  by  a  contained  within  relation  are 
are  not  related  in  the  triplet  set  it  will  not  be  possible  to  parse  the  figure.  The  absence  of 
the  syntactic  relation  between  X  and  Y  from  the  triplet  set 
may  be  because  either  X  or  Y  was  an  isolated  primitive  which  was 
placed  in  a  triplet  to  remove  a  case  of  incompleteness.  For 
example, 


Z 

Y 


X 
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The  graph  for  the  preceding  figure  is, 

\  V 


Syntactically  relating  X  and  Y  results  in  the  graph, 

*R2 


X - ►  Y 


*R5 


The  transformation  rule  which  operates  on  the  triplet  set  for  the  situation  of  the  directed  edges 
having  the  same  terminal  vertex  is, 

(a.0,*Ri).  {y.fi*  Ri) - -(I),  (2).  («,y.*Rj).  where  i/  j. 

1  he  problem  ol  obtaining  all  possible  syntactic  relations  which  can  occur  between  primitives 
contained  within  a  given  primitive  is  not  solved  by  this  formulation.  Though  it  is  only  a  minor 
extension  of  the  above,  and  the  additional  triplets  will  provide  more  parsings,  it  is  not  clear 
at  this  point  that  all  such  possible  parsings  arc  necessary  or  even  desired. 

3.  The  test  for  what  may  be  considered  inconsistent  relationships  must  also  be  applied  to  the 
triplet  set  before  any  attempt  is  made  to  parse  the  figure.  The  relations  are  really  not  incon¬ 
sistent  but  would  appear  so  to  the  parsing  algorithm  when  an  attempt  is  made  to  use  them  and 
so  they  must  be  removed  from  the  triplet  set. 

(i)  The  first  case  of  an  inconsi stent  relationship  between  a  primitive  X  and  a  primitive  Y  can 
be  determined  on  the  directed  graph  of  the  figure  by  a  vertex  X  which  is.  on  the  corresponding 
undirected  graph,  adjacent  (a  and  ft  arc  called  adjacent  vertices  if  s~(n&/^)  for  at  least  one 
edge)  to  a  vertex  Y  where  X  is  also  the  initial  vertex  on  the  dire cted  graph  of  a  directed  edge 
labeled  by  a  contained  within  relation  and  which  has  Z  as  the  terminnl  vertex  and  Y  is  also 
adjacent  to  Z  by  a  directed  edge  which  has  the  same  label  as  the  directed  edge  connecting  X 
and  Y.  For  example. 
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has  the  directed  graph. 


*R1 


The  resulting  directed  graph  is, 


X 


*R5 


Y 


*R5 


*  R  1 
Z 


The  transformation  rule  which  removes  the  triplet  from  the  triplet  set  is. 

ia.fi* Ri),  (fi,y%  *R5).  <aTy>Ri)  - ►  (2),  (3) 

Note  that  the  bar  over  the  triplets  indicates  that  the  order  of  the  primitives  can  be  reversed. 
However,  reversing  the  order  it.  one  of  the  triplets  of  a  rule  necessitates  reversing  the  order 
in  all  of  the  triplets  of  that  rule  which  have  a  bar  over  them.  The  reverse  ordering  of  the  triplets 
will  eliminate  inconsistencies  which  occur  in  a  figure  such  as. 


using  the  transformation  rule, 

(fi.ci* Ri),  R5).  (>'.<i,*Ri)  - ^  (2),  (3) 

(li)  The  second  case  of  an  inconsi  stent  relation  between  X  and  Y  is  determined  on  the  directed 

graph  by  a  vertex  X  which  is  adjacent  to  a  vertex  Y  and  where  X  is  an  initial  node  of  a  directed 

edge  to  a  vertex  Z  which  is  labeled  by  a  contained  within  relation  and  Y  is  an  initial  node  of 

a  directed  edge  to  a  vertex  W  which  is  labeled  by  a  contained  within  relation  and  /  and  V\  are 
not  the  same  vertex,  hoi  example. 
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whieh  has  the  directed  graph, 


/ 


Note  that  the  first  rule  for  inconsistent  relations  will  reduce  the  directed  graph  to, 

/ 


The  second  transformation  rule  for  inconsistent  relations  removes  the  relation  between  X  and 
Y.  This  rule  is, 

(«,P,*R5),  («,y,*Ri),  (/3,5,*Ri),  (y,S,*R5)  —►(!),  (3),  (4) 

In  the  example,  a  =  X,/3  =  Z,y  =  Y  and  8  =  W. 

(iii)  The  third  case  of  an  inconsistent  relation  between  X  and  Y  is  determined  on  the  directed 
graph  by  a  vertex  X,  which  is  the  initial  vertex  for  a  directed  edge  to  vertex  Z  which  is  labeled 
by  a  contained  within  relation  and  X  is  also  the  initial  vertex  for  a  directed  edge  to  vertex  Y 
which  is  labeled  by  a  contained  within  relation  and  Z  is  the  initial  vertex  for  a  directed  edge 
to  Y  which  is  labeled  by  a  contained  within  relation.  For  example,  the  figure 


has  the  directed  graph 


*R  5 


X 


Z 

*R5 
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The  resulting  directed  graph  is. 


X  Z 


and  the  transformation  rule  which  removes  the  triplet  from  the  triplet  set  has  the  same  form  as 
the  rule  before  last,  where  i  -  5. 

(/8 .a ,*R5) ,  (/3,y,*R5),  (y,a,*R5)  - *-(2),  (3) 

The  transformation  rules  are  summarized  in  Appendix  10.2.  They  have  been  created  to 
process  the  triplet  set,  the  final  triplet  set  then  being  transformed  into  a  tree  structure  and 
Reverse  Polish  string.  The  transformation  rules  have  developed  out  of  the  nature  of  the  pic¬ 
torial  data  being  considered  and  the  primitives  and  syntactic  relation  used  in  the  grammar  to 
structurally  describe  the  data.  The  rules  are  not  expected  to  beexhaustive  though  they  will 
serve  a  great  many  configurations  of  the  primitives  and  syntactic  relations  defined  in  the 
previous  chapter.  Once  the  triplet  set  is  considered  complete  the  set  is  partially  ordered. 

5.3.2  PARTIAL  ORDERING  OF  THE  TRIPLET  SET 

Following  the  testing  of  the  triplet  set  for  completeness  and  before  any  tree  structure 
can  be  formed  from  the  triplet  set,  the  triplet  set  must  be  partially  ordered.  The  partial  ordering 
forms  subsets  of  the  triple  set.  which  in  turn  allows  the  tree  structures  or  parsings  to  be  readily 
obtained  from  the  triplet  set.  The  partial  ordering  is  as  follows 

(1)  The  triplet  set  is  scanned  for  a  triplet  which  contains  a  contained  within  relation.  The 
initial  and  terminal  vertices  of  this  triplet  are  subscripted  throughout  the  triplet  set  with  the 
number,  k.  of  the  subset  being  formed.  The  triplet  containing  the  contained  within  relation  is 
then  removed  from  the  triplet  set,  but  must  be  replaced  in  the  triplet  set  if  Step  3  is  not  found 
applicable  to  any  triplets. 

(2)  The  triplet  set  is  now  scanned  for  other  triplets  which  have  the  same  terminal  vertex  as 

the  terminal  vertex  in  Step  1  and  also  a  contained  within  relation.  The  initial  vertex  of  any 
such  triplet  is  subscripted  throughout  the  triplet  set  with  the  number  k.  The  triplet  found  in 
Step  2  with  the  contained  within  relation  is  then  removed  from  the  triplet  set. 

(3)  The  triplet  set  is  now  scanned  for  triplets  which  have  vertices  which  have  a  k  subscript 

except  those  which  have  a  terminal  vertex  which  is  the  same  as  the  terminal  vertex  of  the 
triplet  found  in  Step  1.  These  triplets  are  placed  in  the  kth  subset.  The  vertices  of  any 
triplet  placed  in  the  k,h  subset  are  subscripted  throughout  the  triplet  set  with  a  k  subscript. 

If  a  triplet  which  is  k  subscripted  contains  a  contained  within  relation,  the  vertices  of 
the  triplet  are  dcsubscripted  throughout  the  triplet  set,  and  Step  1  is  repeated  where  k  is  set 
to  k+1.  However,  before  the  k+1  subset  is  formed,  the  k  subset  is  completed. 

(4)  When  no  more  triplets  can  be  added  to  the  k,h  subset,  the  k  +  1  subset  is  formed.  If  there 

is  no  k+1  subset  to  be  formed,  Step  1  is  repeated.  If  Step  1  produces  no  new  triplets  to  be 

subscripted,  the  unsubscripted  vertices  of  the  triplet  set  are  subscripted  with  a  0. 

An  example  of  the  partial  ordering  of  a  triplet  set  is  shown  in  Figure  7.  Once  the  triplet 
set  has  been  partially  ordered  into  subsets,  a  partial  ordering  is  plrced  on  the  triplets  in  the 
subsets.  The  partial  ordering  within  the  subsets  can  be  described  by  the  following: 

(5)  If  three  triplets  in  a  subset  satisfy  either  of  the  two  rules  listed  below  then  the  three 
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(T ,Z,*R  1),  (W,Y,*R2),  (W,Z,*R5).  <X,W,*RS),  (Y,Z,*R5) 

Step  1.  (T.Z,*R1),  (W,Y,*R2),  (W ,  ,Z ,  ,*R5).(X .  W,*R5),  (Y.Z.  +  R5) 
(T,Z,,*ri),  (W ,  ,Y  ,*R2),  (X,W,  ,*R5),  (Y.Z,,*R5) 

Step  2.  (T.Z,,*ri),  (W,,Yr*R2),  <X.W)f*R5>,  (Y,.Z1.*R5) 
(T'Z|.*RI).  (W1.Y,,*R2).  (X,W ,  ,*R5) 

Step  3.  (T.Z,  ,*R1),  (W ,  ,Y  ,  ,*R2),  (X,W,.*R5) 

I _ I 

Subset  1 

Step  1.  (T,Z),*rd,  (Wj.Y  ,  .*R2),  (X2,W2,*R5) 

I - 1  I - 1 

Subset  1  Subset  2 

Step  2,3.  ^ 

Step  4.  (T0,Z,,*R1),  (W2Y  ,  ,*R2),  (X2,W2,*R5) 

I - 1  I _ II _ | 

Subset  0  Subset  1  Subset  2 


Figure  7.  Example  of  Partial  Ordering  of  a  Triplet  Set 
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triplets  are  subscripted  by  the  number  of  the  m**1  such  grouping  in  the  k'*1  subset,  Note  that 
the  triplet  can  be  subscripted  more  than  once  by  these  rules,  where  i,j  *  5. 

(i)  (u.p* Ri),  (a.y* Ri),  (p,y,* Rj) 

For  example. 


(ii)  (u,/-!,*Ri),  (y,p,* Ri),  («,)'. *Rj) 
For  example , 


The  partial  ordering  of  the  triplet  set  into  subsets  is  to  enable  sublevels  of  syntactic 
tree  structures  to  be  formed  easily.  The  ordering  within  the  subsets  is  to  allow  particular 
triplets  to  be  suppressed  while  a  tree  structure  is  being  formed.  Because  the  triplets  are 
eventually  to  be  used,  they  arc  suppressed  but  not  removed  from  the  triplet  set, 

5.3.3  THE  SYNTAX  TREE 

Once  the  triplet  set  has  been  partially  ordered  it  is  transformed  into  a  set  of  binary  tree 
structures.  Each  tree  structure  represents  a  parsing  of  the  figure  where  only  primitives  are 
allowed  to  occur  as  terminal  points  (leaves)  of  the  tree  and  only  syntactic  relations  to  occur 
at  nonterminal  points.  As  indicated  in  5.1,  the  following  discussion  on  the  tree  structure  is 
presented  only  to  illustrate  the  formation  of  the  structural  description.  Section  5. 3. 4  describe s 
the  process  by  which  the  structural  description  in  the  form  of  a  string  can  be  obtained  directly 
from  the  triplet  set. 

The  tree  structure  is  formed  by  a  chaining  process  which  operates  on  linked  triplets. 
Two  triplets  are  considered  linked  if  they  have  a  common  primitive.  In  order  to  form  a  tree 
an  arbitrary  triplet  is  chosen  as  the  starting  point  and  is  written  as  a  binary  tree  with  two 
leaves.  Thus,  the  triplet  forms  the  tree, 


*Ri 


where  the  tree  structure  preserves  the  ordering  of  the  elements  within  the  individual  triplets. 

The  chaining  process  now  replaces  the  leaves  of  the  tree  with  the  syntactic  relations  and 
corresponding  primitives  taken  from  the  triplets  which  are  linked  to  the  triplets  which  compose 
the  tree.  This  process  is  continued  until  the  triplet  set  is  exhausted. 

For  example,  the  figure  and  corresponding  triplet  set  transform  into  the  following  tree 


(T,  Y,*R  1 ) 
(X.Y.*R2) 

(Y,Z,*R2) 
Z 

<6-  Y 
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1  here  is  no  need  to  subscript  the  primitives  of  the  triplet  set  since  no  subsets  will  be  formed. 
The  steps  in  building  the  tree  structure  are  listed,  where  the  (T,Y,*R1)  triplet  is  chosen  first 
in  building  the  first  tree  structure. 


X  Y 


(iii)  *R  l 


Y  Z 


For  completeness,  the  other  five  parsings  are  shown. 

2-  (i)  *R  1  (ii)  *R  1 


Y  Z 


(iii)  *R  1 

/\ 

T  *R2 


*R2  Z 

/  \ 

X  Y 


T  Y 
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The  tree  structures  presented  in  the  example  may  be  considered  as  existing  on  a  single 
level.  However,  the  portion  of  the  tree  representing  the  syntactic  relations  of  a  primitive 
whtch  is  the  initial  vertex  (on  the  directed  graph  representation  of  the  triplet  set)  of  a  contained 
within  relation  or  is  syntactically  related  to  a  primitive  which  is  the  initial  vertex  of  a  contained 
ith in  relation,  drops  to  a  secondary  level.  A  double  line  represents  the  contained  within 
relation  and  annexes  the  secondary  level  in  the  tree  structure.  The  n,h  level  of  the  tree  struc¬ 
ture  can  have  a  substructure  annexed  to  it,  forming  an  n  +  1st  level  .  For  example,  given 


(T.Z,*R1) 

and  the  triplet  (W.Y,*R2) 

set  is,  (W,Z,*R5) 

<X,W,*R5) 
(Y,Z,*R5) 


The  triplet  set  has  been  shown  to  reduce  to  (Tq  Zj  *R1) 
One  parsing  of  the  figure  is  developed  as  follows: 


<W2  ,Y i  ,*R2),  (X2,Wj ,R5). 


(1) 


T 


Z 


T 
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(3)  *R  1 

/\ 

T  7 

\ 


As  indicated  eailier,  the  forming  of  sublevels  in  the  tree  structure  is  the  reason  for  the 
ordering  of  the  triplet  set  into  subsets.  The  partial  ordering  of  the  triplets  within  the  subsets 
is  to  facilitate  the  suppression  of  triplets  while  a  tree  structure  is  being  formed.  The  sup¬ 
pression  of  the  triplets  is  necessary  because  the  triplets  indicated  by  either  of  the  triplet 
subsets  below  (where  i,j^5),  contain  redundant  information. 

(i)  (a.fi* Ri),  (a,y,*Ri),  </S,y,* Rj) 

(ii)  (a./d,*Ri),  (y,/ 9,*Ri),  («,y,*Rj) 

Any  two  of  the  three  triplets  of  (i)  or  (ii)  completely  determine  the  third  triplet.  The  redun¬ 
dancy  is  evident  since  the  third  triplet  is  often  added  to  the  triplet  set  because  a  missing 
relation  is  determined  by  the  completeness  tests.  The  added  triplet  provides  additional  pars¬ 
ings  of  the  figure. 

The  partial  ordering  of  the  triplets  of  the  subsets  which  comprise  the  triplet  set  place 
subscripts  on  the  triplets  which  satisfy  either  of  the  above  subsets.  To  correctly  use  the 
partial  ordering  within  the  subset,  any  two  of  the  three  triplets  may  be  used  in  forming  the  tree 
structure  and  the  third  triplet  is  suppressed.  Since  the  order  in  which  the  two  chosen  triplets 
are  used  determines  different  parsings,  there  are  six  possible  parsings  using  one  of  the  above 
triplet  subsets.  Whether  all  six  parsings  are  distinct  depends  on  the  particular  figure  being 
parsed.  The  six  possible  parsings  of  the  three  triplets  of  triplet  subset  (i)  are  listed  in  Figure 
8.  A  set  of  six  similar  parsings  can  be  obtained  from  the  triplet  subset  (ii).  The  subscript 
on  the  i  and  j  are  to  distinguish  from  which  triplet  each  syntactic  relation  came. 

While  the  arguments  for  the  syntactic  relations  which  were  described  in  Chapter  3  are 
not  being  shown  in  the  examples,  it  is  important  to  note  that  their  values  do  change  as  they 
enter  the  tree  structure.  The  difference  between  two  parsings  may  only  be  the  difference  in 
value  of  the  arguments  of  a  single  syntactic  relation  appearing  in  the  tree  structure.  Parsings 
(1)  and  (3)  of  the  above  example  illustrate  this  fact.  The  difference  between  parsings  (2)  and 
(4)  are  also  only  the  difference  in  the  valcc  of  the  arguments  of  *R'\ap  in  (2)  and  *Ri„v  in  (4). 

The  syntactic  relations  found  in  the  triplets  transfer  directly  to  the  tree  structure  but 
the  value  of  their  arguments  do  not.  The  arguments  change  in  \alue  because,  while  the  higher 
level  constituents  are  being  related  by  the  same  syntactic  relations  which  originally  related  the 
primitives  of  the  figure,  the  higher  level  constituents  have  different  dimensional  values  than 
the  primitives  and  hence  require  different  values  for  the  arguments  of  the  same  syntactic 
relations. 

The  obtained  syntax  tree  can  be  considered  a  rooted  tree.  A  rooted  tree  is  a  tree  in  which 
one  node,  called  the  root,  is  given  a  special  significance.  This  introduces  a  direction  in  the 
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Figure  8.  Possible  Parsings  of  a  Triplet  Subset 
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tree,  away  (or  in  this  case  down)  from  the  root  and  towards  (up)  the  root.  If  a  tree  is  rooted 
each  node  has  a  node  immediately  above  it,  unless  the  node  is  the  root  of  the  tree.  In  addition, 
because  there  is  a  specified  order  of  the  lines  around  any  node,  the  tree  is  considered  ordered 


■t 

where  the  nodes  are  ordered  from  left  to  right.  A  property  of  the  node  to  be  used  later,  called 
the  outer  degree  of  the  node,  can  be  determined  by  drawing  paths  from  the  root  to  the  leaves  Of 
the  tree.  This  uniquely  associates  a  direction  with  each  arc.  The  outer  degree  of  a  node  x  is 
independent  of  the  orientation  of  the  tree. 

5.3.4  THE  REVERSE  POLISH  STRING 

The  tree  structure  is  not  the  desired  final  form  of  the  structural  description.  It  will  be 
converted  to  a  linear  string  which  uses  Reverse  Polish  notation  to  provide  a  more  useable  form 
of  the  structural  description  to  the  semantic  component.  As  pointed  out  earlier,  the  explicit 
formation  of  the  tree  structure  is  not  a  necessary  step  in  order  to  obtain  the  Reverse  Polish 
string. 

The  tree  structure  can  be  readily  transformed  into  a  linear  string  which  uses  Reverse 
Polish  notation  to  order  the  constituents  and  the  syntactic  relations  between  them.  The  string 
can  be  written  by  traversing  a  path  around  the  tree  structure  from  left  the  right.  The  string  is 
written  from  left  to  right,  where  primitives  are  pulled  off  as  they  occur  and  syntactic  relations 
are  placed  in  the  string  at  their  last  possible  encounter  with  the  path.  The  one  exception  is  in 
leaving  a  sublevel,  in  which  case  the  contained  within  relation  is  entered  into  the  Polish  String 
after  a  primitive  of  the  next  higher  level  is  entered  into  the  string.  Recall  that  the  contained 
within  relation  is  represented  in  the  tree  structure  by  a  double  line. 

An  example  should  clarify  any  confusion  on  this  process.  Given  the  figure. 
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The  graph  and  triplet  set  is: 


( T ,  D ,  *  R 1 ) 

(A,l),*R5)  "" 

(B,D,*R5) 

(t',D.*R5)  Subset  1 

(A,B,*R2) 

( A ,  C ,  *  R 1 ) 


One  possible  parsing  of  the  figure  is. 


Hie  Reverse  Polish  string  of  this  parsing  is  <bui!t  up  from  the  dotted  path  giving, 

T,A,C,*R1,B,*R2.D,*R5.*RI. 

At  this  time  no  attempt  is  made  to  obtain  all  possible  parsings.  Otherwise,  a  triplet  relating 
B  and  C  would  be  needed  though  it  would  be  suppressed  in  this  particular  parsing.  As  men¬ 
tioned  earlier,  it  is  not  clear  whether  every  possible  parsing  is  needed  for  pattern  recognition 
or  whether  producing  every  parsing  is  merely  a  waste  of  computer  time.  The  number  of  different 
parsings  which  can  be  semantically  interpreted  by  the  program  depends  on  the  degree  of  exten¬ 
siveness  of  the  grammar. 

In  order  to  obtain  the  syntactic  string  directly  from  the  triplet  set,  the  chaining  process 
which  operates  on  linked  triplets  is  used.  The  chaining  process  replaces  a  primitive  in  the 
string  with  the  syntactic  relation  and  corresponding  primitives  of  a  triplet  which  is  linked  to 
the  triplet  containing  the  primitive  being  replaced.  As  is  shown  in  forming  the  tree  structure, 
partial  ordering  and  suppression  of  triplets  in  the  triplet  set  are  necessary. 

Using  the  example  on  this  page,  the  steps  in  forming  the  syntactic  string  are  as  follows: 

(1)  T,D,*R  1 

(2)  T,A,D,*R5,*R1 

(3)  T,A,B,*R2.D,*R5,*R1 

(4)  T.A.C,*R1  ,B.*R2.D,*R5.*R1 
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In  order  to  test  the  well-formedness  of  a  syntactic  string  in  Reverse  Polish  form  the 
procedure  outlined  below  is  followed, 

(it  Assign  a  weight  W  to  each  element  of  the  string.  The  weights  are  assign¬ 
ed  by  the  formula  W  1  d,  where  d  is  the  number  of  lines  leaving  a  node 
(away  from  the  root  as  defined  in  5.3,3). 

(ii)  Find  £W(i)  for  i  1 . n  elements  of  the  string,  where  the  left  most 

element  of  the  string  is  i=l. 

n  J 

(iii)  Well-formedness  requires  that  XW(i)  l  and  lW(i)  for  any  i  •  n  is  never 

J. 

less  than  1,  i.e.,  SW’(i)  5  1,  j  1 . n, 

i-  1 

Testing  the  well-formedness  of  the  syntactic  string  of  the  previous  example: 


T 

A 

C 

*R1 

B 

*R2 

D 

*R5 

*P1 

i  : 

1 

-> 

3 

4 

5 

6 

7 

8 

9 

W(i) : 

1 

1 

1 

-1 

1 

-1 

-1 

1 

-1 

'iW(i) ;  1  2  3  2  3  2  1  1  1 

T  9 

Since  iW(i)  1  for  all  j  1 . 9  and  2£ W ( i )  1,  the  siring  is  well-formed.  Note  that  the  degree  d 

i  i 

of  *R5  in  the  tree  structure  is  0  and  the  degree  of  the  primitive  D  is  2. 

5.4  A  SUMMARY  OF  THE  SYNTACTIC  COMPONENT 

The  syntactic  string  will  now  be  sent  to  the  semantic  component  for  a  semantic  inter¬ 
pretation.  Because  the  semantic  component  is  composed  of  a  grammar  which  operates  on 
linear  strings  of  constituents,  the  picture  which  is  a  two-dimensional  configuration  must  be 
reduced  to  a  one-dimensional  configuration.  This  reduction  is  performed  by  the  syntactic 
component  which,  in  summary  obtains  the  structural  description  in  the  form  of  a  linear  string  by 
using  the  grammar  Kl*  to  form  the  lines  in  the  figure  from  the  discrete  points  and  then  form  the 
higher  level  primitives  from  the  lines.  Because  the  syntactic  relations  are  contained  implicitly 
in  the  pictures  but  must  be  made  explicit  in  the  syntactic  strings  for  the  semantic  component, 
the  syntactic  component  develops  the  triplet  set  of  3-tuples  which  are  primitives  and  their 
corresponding  binary  relations.  This  triplet  set  is  preprocessed  and  then  normalized  for  the 
semantic  component  by  writing  it  as  a  linear  string  using  Reverse  Polish  notation. 

The  transformation  of  the  triplet  set  actually  results  in  a  set  of  structural  descriptions. 
These  multiple  parsings  can  be  the  result  of  two  different  processes. 

(1)  Because  there  is  a  choice  as  to  which  artificial  lines  to  add  to  a  figure 
to  allow  the  figure  to  be  parsed,  each  different  configuration  of  the  figure 
after  the  artificial  lines  have  been  added  provides  a  different  triplet  set. 

(2)  In  turn,  each  triplet  set  can  provide  multiple  parsings  of  the  figure,  as 
described  earlier  in  this  chapter. 

Either  of  the  above  steps  can  cause  the  number  of  parsings  of  a  figure  to  be  quite  large. 

The  resulting  syntactic  descriptions  are  sent  to  the  semantic  component,  the  analyses 
of  which  are  described  in  the  next  chapter. 
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6.  THE  SEMANTIC  COMPONENT  OF  SAP 

6.1  A  SEMANTIC  INTERPRETATION 

The  semantic  component  operates  on  the  structural  description  L*^  to  provide  a  semantic 
interpretation  to  the  pictorial  data,  using  a  grammar  to  provide  this  interpretation.  As  indicated 
in  the  previous  chapters, a  one-dimensional  grammar  operates  on  linear  strings.  For  this  reason 
the  syntactic,  component  transforms  the  input  picture  L|  to  a  syntactic  string  L*^  which  struc¬ 
turally  describes  L..  The  grammar  of  the  semantic  component  operates  on  the  syntactic  string, 
eventually  placing  it  into  one  or  more  classes.  The  classification  of  the  syntactic  string  is 
actually  the  assigning  to  it  the  names  (labels)  of  the  classes  into  which  it  is  placed.  The 
forming  of  higher  level  constituents  from  the  symbols  of  the  syntactic  string  is  defined  by  the 
rule  stated  in  Chapter  4  for  forming  segments  of  syntactic  strings.  This  rule  is, 
<sss>::=<primitive> »  primiti  ve>  <syntactic  re  1  a tion>  | 

<sss><pr  m  tive><syntactic  relation''  | 

<primitivt <sss">  < syntactic  relation>  | 

<sss>  <sss>  <syntactic  relation> 

This  is  partially  represented  by  a  tree  structure  in  Figure  9.  The  semantic  component,  in 
using  the  grammar,  will  assign  names  of  classes  to  the  constituents  which  are  genetically 
represented  by  <sss>  in  the  tree  of  Figure  9. 


<sss> 


Figure  9.  General  Syntax  Tree 

Before  desc  ibing  the  grammar  which  comprises  the  semantic  component,  it  is  worthwhile 
to  first  furthei  discuss  the  syntactic  description  which  the  semantic  component  receives.  It 
was  shown  in  the  last  chapter  that  the  Reverse  Polish  string,  which  is  the  syntactic  description, 
is  obtained  from  a  tree  structure  which  is  also  a  syntactic  description  of  the  figure  to  be 
semantically  interpreted. 
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Using  the  example  of  the  directed  graph. 

<primitive>p^ 

<syntactic  relation>K 
s  1 


<primitive>p 


<syntactic  relation>, 


<primitive>p^ 


the  tree  structure  obtained  by  the  syntactic  component  representing  one  parsing  of  the  figure 
is  of  the  form  shown  below. 


The  important  point  is  that  the  tree  structure  immediately  above  is  isomorphic  to  the  tree  struc¬ 
ture  shown  below, 


<sss> 


the  general  form  of  which  is  presented  in  Figure  9.  However,  the  <sss>  constituents  in  the 
above  tree  structure  remain  unlabeled  by  the  syntactic  component.  It  is  the  semantic  component 
which  will  assign  labels  to  the  <sss>  constituents. 

The  algorithm  described  in  Section5.3.3for  creating  tree  structure  of  the  form  shown  on 
this  page  could  be  easily  applied,  with  only  slight  modification,  to  create  the  tree  structure 
above  from  the  triplet  set.  Also,  the  algorithm  described  in  Section  5.3.4  for  forming  the  Reverse 
Polish  string  from  a  tree  structure  directly  applies  to  the  second  tree  structure  above. 
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In  view  of  this,  it  is  somewhat  arbitrary  as  to  which  form  of  tree  structure  is  chosen  to  repre¬ 
sent  the  parsing  of  the  figure.  The  second  structure  has  the  advantage  of  explicitly  representing 
the  constituents  which  are  to  be  labeled  by  the  semantic  component.  However,  in  this  teport 
the  first  tree  structure  on  the  last  page  is  used  because  it  is  a  more  concise  notation  by  impli¬ 
citly  representing  the  structural  constituents. 

The  grammar  of  the  semantic  component  K3*  is  a  5-tuple  (T3.N3.R3.C3.SC).  The  set 
T3  is  the  set  of  terminal  symbols  and  is  a  union  of  the  set  P  of  primitives  and  the  set  S  of 
syntactic  relations.  The  set  N3  is  the  set  of  nonterminal  symbols  of  the  grammar.  The  set  R3 
is  the  rules  of  the  grammar  which  determine  which  generic  label  should  be  assigned  to  a 
particular  constituent.  The  set  C3  is  a  set  of  matrices  which  contain  constraints  for  the  rules 
of  R3.  The  goal  of  the  grammar  K3*  is  a  scene.  SC.  and  does  not  appear  on  the  right  hand 
side  of  ant  of  the  rules  of  R3.  The  grammar  K3*  is  partitioned  into  two  levels,  each  level 
consisting  of  rules  from  R3  which  assign  meaning  to  the  figures  of  the  scene. 

The  first  level  of  the  grammar  consist  of  rules  which  assign  meaning  in  terms  of  the 
structure  of  each  individual  figtre.  The  second  level  accepts  those  structural  descriptions 
which  have  not  received  a  singular  semantic  interpretation  from  the  first  level  of  the  grammar. 
The  figures  being  sent  to  the  second  level  of  the  grammar  may  be  ambiguous  at  this  point, 
that  is,  more  than  one  meaning  has  been  assigned  to  the  figure.  However,  it  is  just  as  likely 
that  the  figure  canny  be  completely  identified  by  the  first  level.  The  second  level  attempts 
to  semantically  inteipret  the  figure  in  terms  of  other  figures  in  the  scene.  Thus,  the  second 
level  uses  the  contextual  surroundings  of  the  figure,  i.e.,  the  syntax  of  the  scene,  to  assign 
meaning,  while  the  first  level  uses  the  syntactic  structure  of  the  figure,  i.e.,  the  contextual 
surroundings  of  the  primitives  to  assign  meaning  to  the  figure.  Contextual  constraints  are 
considered  at  both  levels  of  the  semantic  component  grammar.  In  addition  it  is  desired  that 
the  second  level  grammar  use  a  form  of  deductive  inference  decision  making.  The  subcompo¬ 
nents  of  the  semantic  component  are  shown  in  Figure  10. 

To  effectively  use  the  first  and  second  level  grammars,  the  syntactic  string  is  first 
processed  by  an  abstracting  subcomponent.  The  abstracting  process  described  in  the  next 
section  obtains  an  abstraction  of  the  figure  to  be  identified.  This  abstraction  eliminates  many 
fruitless  attempts  by  the  first  level  grammar  to  assign  a  semantic  interpretation  to  the  figure. 

6.2  THE  ABSTRACTION  OF  FIGURES 

To  effectively  use  the  rules  of  the  first  level,  an  abstraction  of  the  structural  description 
is  obtained  to  provide  a  preliminary  classification  of  the  figure  without  interference  from  details 
of  the  figure.  The  preliminary  classification  eliminates  the  searching  through  a  great  deal  of 
the  grammar  which  would  not  be  applicable.  Thus,  the  first  level  of  the  grammar  would  be 
partially  ordered  with  respect  to  general  classifications  of  the  graphic  data.  It  is  reasonable 
to  consider  all  data  to  be  amendable  to  some  form  of  classification  based  on  major  character¬ 
istics  of  the  expected  figures. 

The  abstraction  subcomponent  receives  from  the  syntactic  component  the  Reverse  Polish 
string  representing  the  structural  description  of  the  scene.  Thus .  the  first  task  of  the  abstraction 
subcomponent  is  to  obtain  the  individual  figures  of  the  scene  to  enable  the  first  level  grammar 
to  operate  on  these  syntactic  string  segments  independently.  To  obtain  the  syntactic  string 
segments  of  the  scene  a  modified  bottom- up  strategy  is  used  on  the  syntactic  string  operating 
on  the  string  from  right  to  left. 


-65- 


t 

I 

I 


1  •’ i guru  10.  The  Seniiinlie  Component 


-66- 


The  syntactic  string  is  segmented  into  the  various  tigures  according  to  the  grammar  rule 
for  forming  syntactic  string  segments.  This  rule  and  two  others  used  to  ibtain  the  individual 
figures  are  listed  beiow. 


<scene'>  .j  : :=  C'igurex .  |  <scene^. .  <scene>, . 

^syntactic  relation‘>/*Ri(n1  ^0,n2,n3) 
for  i  =  l  ,2 ,3  or  4 

<figure^  jj  : :=  <sss'>.. 

<sss>  j|  ::=<sss>..  sssXj  '-syntactic  relation>  | 

<sss^(.  <primitive>. .  < syntactic  relation  ^  | 

<primitive''i.  <sss>,.  <syn tactic  relation>  | 

<prinmi ve^j. <primitive^( . <syntactic  relation> 

The  i  and  j  subscripts  on  the  constituents  will  be  explained  shortly.  The  restriction 
on  the  production  rule  for  a  scene  indicates  that  a  figure  is  defined  to  be  either  of  the  following: 

(1)  A  figure  X  is  represented  by  a  syntactic  string  segment  sss,  and  is 
related  to  a  figure  Y  represented  by  a  syntactic  string  segment  sssj  by  a 
directional  relation  (horizontal  or  vertical  relation)  *Ri(n,  ,n2  ,n3)  where 
i  =  l,2,3  or  4  and  n^o.  Since  n,=()  signifies  the  two  figures  X  and  Y 
are  touching,  n,  0  requires  X  and  Y  to  be  two  related  constructions  in  a 
picture  which  have  no  contiguous  elements.  As  defined  in  Chapter  4, 
n2  and  n3  indicate  the  secondary  position  between  constituents,  where 
n3=0  and  is  not  present  for  a  directional  relation. 

(2)  A  figure  X  is  represented  by  a  syntactic  string  segment  sss,  and  is 
related  to  a  figure  Y  represented  by  a  syntactic  string  segment  sss2  by  a 
contained  within  relation,  *Ri(n ,  ,n2.n3)  where  i=5  or  b. 

The  two  above  definitions  allow  figures  to  be  drawn  in  a  completely  recursive  manner. 
Thus,  the  first  definition  allows  any  number  of  non-contiguous  figures  in  a  horizontal  row, 
vertical  column,  or  combination  of  the  two  to  be  structurally  represented  in  a  syntactic  string 
and  the  abstracting  component  will  be  able  to  segment  the  string  into  the  individual  non¬ 
contiguous  figures.  Similarly,  the  second  definition  allows  any  number  of  figures  to  be  imbedded 
within  a  figure.  For  example,  the  use  of  the  two  definitions  allows  a  row  of  houses  to  be 
imbedded  within  a  house,  as  shown  below. 
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lf  it  is  desired  to  not  allow  figures  to  be  imbedded  within  figures,  it  is  necessary  only  to 
require  the  syntactic  relation  which  occurs  on  the  right-hand  side  of  the  rule  defining  <scene> 
to  be  a  directional  relation  and  eliminate  the  second  definition  ol  above. 

Once  the  syntactic  string  has  been  segmented  into  the  segments  representing  the  figure 
of  the  scene,  the  abstracting  subcomponent  then  abstracts  the  syntactic  string  segment  to 
obtain  the  basic  features  of  the  figure.  The  abstracting  process  in  effect  first  removes  all 
detail  Trom  the  figure.  This  leaves  an  outline  of  the  figure  which  is  further  reduced  to  a  set 
ol  abstractions  of  the  figure,  each  abstraction  somewhat  more  abstract  than  the  last.  In  attempt¬ 
ing  to  generally  categorize  the  ligure,  the  abstractions  are  used  in  the  reverse  order  of  which 
they  were  produced,  the  most  abstract  first.  The  method  for  performing  the  abstractions  is 
actually  a  heuristic  which  is  independent  of  any  grammar  being  used  to  semantically  interpret 
the  figure. 

To  obtain  the  abstractions  of  the  figure  from  the  structural  description  of  the  figure,  as 
in  obtaining  the  string  representing  the  individual  figure,  a  bottom-up  strategy  is  used  on  the 
syntactic  string.  The  rule  to  perform  this  abstracting  is  the  same  rule  shown  above  lor  forming 
the  syntactic  string  segment  constituents.  The  i  and  j  are  attributes  of  the  <sss:>  constituent 
which  allow  the  abstractions  to  take  place.  As  the  vsss>  constituents  are  formed,  they  are 
assigned  the  name  of  the  predominant  constituent  of  those  constituents  which  form  the  sss>  , 
This  predominant  constituent  is  the  one  which  has  the  largest  area.  Thus,  the  i  subscript  on 
the  sss  '  constituents  is  the  name  of  the  predominant  subconstituent,  where  i  may  be  R 
Tl,  etc..  1  he  value  of  i  is  then  used  to  name  higher  level  constituents  when  the  particular 
constituent  is  considered  the  predominant  constituent.  The  j  is  the  area  of  the  constituent 
and  is  used  to  determine  which  constituent  is  the  predominant  constituent  in  forming  a  higher 
level  constituent. 

The  following  example  should  clarify  the  process  or  abstracting  a  single  figure.  The 
syntactic  string  segment  constituents  are  represented  in  the  string  by  Sn(i.k.nt)  which  is  defined 
as,  n  =  segment  constituent  number:  i=l  if  the  predominant  constituent  is  a  rectangle,  i  =  2 
if  the  predominant  constituent  is  an  isosceles  triangle:  k  =  horizontal  dimension:  m  =  vertical 
dimension.  Step  I  is  the  syntactic  string  of  the  figure. 

I  •  R(3,2,2),R(4,2,4),*R4,R(5,2,2),*R4,TI(l,8,6).*Rl,R(2,4,5),R(l,8,12),*R5(BOT),*Rl 

2.  R(3.2,2),Rt4,2,4),*R4,R(5,2,2),*R4.TI(l,8,6),*Rl,SI(l,8,12),*RI 

from  Sl(l, 8. 12)  — *R(2,4,5),R(  1 ,8, 1 2),*R5(BOT) 

3.  S2(l  ,4,4),R (5,2,2),*R4,TI(  1 ,8,6),*R  I  ,S1(1.8, 1  2),*R  1 

from  S 2 ( 1,4,4)  — ►  R(3,2,2),R(4,2,4),*R4 

4.  S3(l  ,6,4),TI(  1 ,8,6),*R  1  ,SI  ( 1 ,8, 12) ,*R  1 

from  S2(  1 ,6,4)  — ►  S2(l  ,4,4). R( 5,2,2), *R4 

5.  S4(2,8, 1()),S  I  ( 1 ,8,1 2),*R  1 

front  S4(2,8. 10)  — *  S3(  1 ,6,4),TI(1 ,8,6),*R  I 

6.  S5( 1 ,8,22) 


from  S5(  1 ,8,22)  — *  S4( 2(8 . 10),S  I  ( 1 ,8, 1 2).*R  I 
hooking  at  the  abstractions  pictorially. 
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xamining  the  abstractions  in  their  reverse  order,  that  is  the  most  abstract  first,  the  fifth 
abstraction  or  Step  5  would  indicate  that  the  figure  should  be  operated  upon  by  the  first  level 
grammar  and  considered  to  be  in  a  class  which  represents  the  category  of  house  type. 

The  six  categories  into  which  the  pictorial  data  being  considered  in  this  report  may  be 
Placed  after  it  has  been  abstracted  to  a  string  composed  of  the  three  ordered  constituents, 
.syntactic  string  segment  -,  Syntactic  string  segment,  Syntactic  relation>  are  listed  below. 


Rectangle  contained  within  rectangle  and 
the  final  abstraction  is  a  rectangle. 


Triangle  above  rectangle,  where  3*  (area 
of  triangle)  -  (area  of  rectangle)  and  the 
final  abstraction  is  a  rectangle. 


3.  SILO. 


Triangle  above  rectangle,  where  3*  (area 
of  triangle)  <  (area  of  rectangle)  and  the 
final  abstraction  is  a  rectangle. 


-69- 


4.  TREK; 


r  r  •  a  n  g  1  c  above  rectangle,  where  (area 

of  triangle)  '  (area  of  rectangle)  and 
the  final  abstraction  is  a  triangle, 


5.  BULBOUS: 


Circle  above  rectangle,  and  the  final  ab- 
straetion  is  a  circle  or  rectangle. 


6.  OFFICE  BLDG: 


Rectangle  above  rectangle,  and  the  final 
sbstraction  is  a  rectangle. 


7.  UNICORN:  Anything  which  cannot  be  placed  into  one  of 

the  first  six  categories. 

Because  the  segmenting  of  the  syntactic  string  into  segments  which  represent  individual 
figures  in  the  scene  and  the  abstracting  process  of  the  figures  both  use  a  bottom-up  strategy, 
the  two  processes  can  be  performed  at  the  same  time.  Thus,  while  a  scene  is  divided  into 
the  figures  which  comprise  it,  the  figures  are  being  abstracted.  The  use  of  the  bottom-up 
strategy  avoids  the  left  recursion  problem  of  the  top-down  strategy  described  in  Chapter  2. 
For  example,  consider  a  scene  composed  of  the  two  figures  below. 


The  abstraction  subcomponent  would  operate  on  the  syntactic  string  to  obtain  the  tree 
structure  shown  on  the  next  page.  The  tree  structure  indicates  the  individual  figure  and  its 
abstractions.  As  defined  earlier,  the  j  subscript  on  a  constituent  is  its  surface  area.  The 
relations  and  primnives  do  not  have  their  arguments  shown.  The  relation  *R4^ 


repre  sents 
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the  restricted  relation  *Ri(n ,">0 ,n2,n3)  previously  discussed  in  this  chapter.  Again,  .ne  n , >  1 
indicate  s  that  the  figures  are  not  contiguous  which  in  turn  allows  the  abstracting  component 

to  separate  them. 


■  rj  litre 
I  *1 


<syntactic 

relation'’ 


*R46 


The  entire  abstracting  subcomponent  has  been  implemented  in  FORTRAN  on  the  IBM  7072. 
The  program  is  described  and  listed  in  Appendix  10.1  of  this  report. 

The  abstracting  process  is  a  valuable  tool  in  dealing  with  the  problem  of  multiple 
parsings.  If  all  of  the  parsings  must  be  operated  upon  by  the  semantic  component,  the  abstract¬ 
ing  process  should  have  a  great  deal  of  time  in  eliminating  some  unproductive  attempts  by  the 
first  level  grammar.  The  multiple  parsing  problem  is  also  the  reason  for  the  semantic  compo¬ 
nent  being  divided  into  two  levels.  The  division  allows  the  syntactic  string  to  be  semantically 
interpreted,  one  segment  (which  represents  one  figure)  at  a  time.  If  each  figure  was  not  pro¬ 
cessed  separately,  because  of  the  large  number  of  parsings  per  figure,  the  computer  could 
possibly  run  out  of  space,  or  use  an  excessive  amount  of  time.  For  if  a  scene  of  two  figures 
which  have  m,  and  m2  parsings  respectively,  the  maximum  number  of  passes  through  the  seman¬ 
tic  component  1st  level  grammar  to  identify  the  figures  in  a  scene  is  m,  +  m2.  However,  the 
the  number  of  passes  through  the  semantic  component  grammar  if  the  entire  scene  must  be 
considered  at  each  pass  is  m,  x  m2.  The  fact  that  the  abstracting  process  is  a  heuristic 
independent  of  the  grammar  rules  which  it  uses  may  prove  to  be  an  additional  nicety  of  the 
procedure,  allowing  various  criterion  other  than  largest  area  to  determine  the  predominant 
constituents. 

The  abstracting  process  provides  a  Gestalt  of  the  figure  and  hence  plays  an  important 
role  in  human  pattern  recognition.  M.D.  Vernon42  points  out  that  the  most  important  feature 
in  human  pattern  iecognition  is  the  general  outline  or  contour  of  the  object.  This  may  be 
because  a  child  learns  first  by  touch  and  thus  associates  objects  with  their  contour.  In  any 
case  there  is  little  doubt  that  an  adult  tries  to  first  classify  a  figure  generally  before  he 
proceeds  to  identify  it.  Few  individuals  would  try  to  identify  a  figure  as  a  church  if  they 
didn't  think  that  the  figure  is  some  type  of  building.  Another  manner  in  which  psychologists 
might  describe  the  abstracting  process  is  providing  a  set.  That  is,  the  details  of  the  figure 
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make  sense  only  because  we  know  what  type  of  details  to  expect  by  means  of  a  set  orientation 
(no  connection  to  set  theory).  The  abstracting  process  allows  this  orientation  to  take  place. 

A  significant  use  of  the  abstracting  process  is  in  the  filtering  of  a  complex  scene.  For 
example,  if  aerial  photographs  are  to  be  scanned  for  a  particular  figure,  the  abstracting  process 
would  quickly  eliminate  those  figures  from  the  scene  which  are  of  no  interest.  This  avoids  the 
brute  force  approach  of  attempting  to  identify  every  figure  in  the  photograph.  Of  course,  the 
choice  of  criterion  by  which  to  perform  the  abstracting  is  critical. 

Once  the  abstracting  process  is  completed,  and  by  means  of  one  or  more  of  the  abstrac¬ 
tions  the  figure  has  been  roughly  classified,  the  original  structural  description  (the  complete 
syntactic  string  segment)  is  operated  upon  by  the  first  level  of  the  grammar. 

6.3  THE  FIRST  LEVEL  GRAMMAR  OF  THE  SEMANTIC  COMPONENT 

As  pointed  out  in  Chapter  2,  the  primitives  have  no  meaning  independent  of  their  use  in 
a  particular  environment.  This  problem  has  not  arisen  in  the  work  by  Narasimhan  because 
the  primitives  are  lines  as  are  the  graphic  data.  Hence,  except  for  noise,  if  a  primitive  is  pre¬ 
sent  it  is  known  to  be  a  bubble  chamber  track.  The  severe  semantic  constrictions  on  his  problem 
provide  the  primitives  with  a  singular  meaning.  But  in  a  less  semantically  constricted  problem 
where  the  primitives  may  have  one  of  several  meanings,  a  major  difficulty  atises.  The  problem 
is  that  a  single  primitive,  which  is  able  to  have  multiple  meanings,  can  be  assigned  a  meaning 
only  in  the  context  of  other  primitives.  For  example,  a  rectangle  cannot  be  identified  as  either 
a  door  or  a  window  until  its  location  and  dimensions  relative  to  the  surrounding  primitives 
are  considered. 

In  view  of  this  difficulty,  many  of  the  rules  of  the  first  level  grammar  of  the  semantic 
component  are  context  sensitive,  providing  restrictions  which  must  be  satisfied  before  the 
rules  are  considered  applicable.  These  restrictions  are  generally  concerned  with  the  relative 
size  or  location  of  the  constituents  of  the  rule.  The  rules  are  of  the  general  form, 

X::=Y,W,*Ri,Z,*Rj,..,/0 

0,=  jF<V  0  G(xm) 

<  xk >xm  ■  *Rj(n i  ,n2,n3) 

where  the  arguments  of  the  constituents  are  not  shown.  Note  that  the  right  hand  side  of  the 
rule  is  written  in  early  Reverse  Polish  notation.  The  rule  states  that  constituent  X  may  be 
formed  from  constituent  Y  in  relation  Ri  with  constituent  W.  this  in  relation  Rj  with  constituent 
Z,  etc.,  within  the  context  of  0. 

The  0  is  a  set  of  contextual  constrictions  {0,}  such  that  the  members  of  0  may  be  of 
either  form  indicated. 

1.  When  >r,=  F(xk)  0  G(xJ,  F  and  G  are  functions  which  obtain  the  height 
(II)  or  width  (W)  of  the  constituent  xk  and  xm  where  k  may  equal  m. 

Ihe  .0  is  a  member  of  the  set  of  arithmetic  relations.  For  example, 
if  F=H,  and  xk  =  R  (a,b,c)  then  F(xk)=H(R(a,b,c))=  c. 

2,  When/ff|=xk,xm ,  *Rj  (ni,n2,  n3)  the  j,  n3,  n2,  and  n3  may  be  constrained 
values  indicating  restrictions  on  the  syntactic  relation  between  the 
constituents  x^  and  xm.  The  definitions  of  n^,  n2,  and  n3  are  found 
in  Chapter  4. 

Thus  the  constraints  may  operate  on  a  single  constituent  or  relatively  between  two 
constituents.  The  use  of  the  contextual  constraints  raises  a  major  problem.  The  problem  is 
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tliat  a  single  constraint  may  not  necessarily  operate  on  constituents  ol'  a  single  rule  but  across 
rules  on  constituents  at  different  levels.  This  requires  that  the  formulation  of  the  restrictions 
be  defined  and  noted  independently  of  the  formulation  of  the  rules. 

To  accomplish  this  the  constraints  are  written  as  a  set  of  matrices.  C3.  Each  matrix 
contains  the  constraints  for  different  constituents  which  are  restricted  by  a  common  constraint. 
It  is  then  possible  for  the  rules  to  be  flagged  with  the  entries  in  the  matrix  which  pertain  to 
the  constituents  of  the  rules  which  are  to  be  constrained.  While  the  constraints  are  between 
constituents  they  actually  determine  the  identification  or  well-formedness  of  a  figure,  depending 
on  whether  the  grammar  is  being  used  for  recognition  or  generation.  Since  it  is  the  figure  for 
which  the  constraints  are  ultimately  operating  and  because  a  top-down  strategy  is  used  by  the 
first  level  of  the  semantic  component,  the  matrix  entries  which  apply  to  the  constituents  of  a 
figure  are  attached  to  the  rule  which  has  the  name  of  the  figure  on  the  left-hand  side  of  the  rule. 

An  example  should  clarify  the  preceding  paragraph.  Consider  a  grammar  which  wishes 
to  be  able  to  recognize  either  of  the  two  structures  shown  in  their  general  form. 


Assume  that  the  structures  can  be  identified  as  either  a  doghouse,  shed,  or  garage 
depending  upon  how  they  satisfy  various  constraints.  To  perform  this  recognition  the  following 
grammar  is  used.  It  should  be  noted  that  the  use  of  the  grammar  is  to  assign  labels  to  the 
constituents  of  the  parsing  of  the  figure.  Thus,  the  semantic  grammar  is  essentially  assigning 
labels  to  the  <sss>  constituents  defined  earlier,  the  assignment  algorithm  being  performed  in 
a  top-down  manner. 


<type  1  bldgN::=<doghouse>  |  <shed>  |  <garage> 

<doghouse>. :=  <facade  1>  /  ( 1 , 1),(1 ,2), (2, 2). 

(2. 3) , (3, 2), (3, 3) 

<  shed>::=<facade  1>  /  (1 , 1), (1,2), (2, 2), 

(2. 3) , (3, 2). (3  3) 

<garage>: : ^<facade  1  >  /  ( 1 , 1),(1 ,2), (2, 2), 

(2. 3) , (3, 2), (3, 3) 

facade  1>::=  >roof>  <front  Ixvertical  relation  1> 

<front  1>::=  <panel> 

<door>  <panel">  <contained  within  relation  5> 
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<rool>:  :=<isosceles  triangles 
<panel  : :=<rectangle 
door>:  :-<rectangle  :> 

<rectangle>::  R(n,h,v) 

<isosceles  triangle'-::  Tl(n,h,v) 

The  contextual  restrictions  are  indicated  by  the  matrix  entries  (i.j),  which  in  this  case 
apply  to  the  type  1  hid g  constraint  matrix.  The  rows  and  columns  of  the  matrix  are  labeled 
with  the  names  of  the  constituents  which  are  restricted  by  a  constraint  ()..  The  names  of  the 

rows  and  columns  of  the  constraint  matrix  and  the  corresponding  number  of  the  rows  and  columns 
are : 

roof  -  I 
panel  =  2 
door  3 

constraint  matrix  is  shown  in  Table  2.  The  entries  in  the  matrix  may  be  defined 

For  entry  (i,j),  it  i  j,  then  the  constraint  involves  only  one  constituent 
and  is  constraining  the  constituent's  height  relative  to  its  width. 

For  entry  (i.j).  if  i<j .  then  the  constraint  involves  two  constituents  and 
is  constraining  some  combination  of  their  heights  and  width. 

For  entry  (i.j).  if  i  j.  then  the  constraint  involves  the  syntactic  relation 
between  two  constituents  and  is  constraining  their  relative  position  in 
one  or  both  directions. 

The  row  and  column  labels  are  those  constituents  which  appear  on  both  the  left-hand  side  and 
the  right-hand  side  of  the  rules.  In  addition,  constituents  which  are  not  explicitly  constrained 
need  not  be  assigned  a  row  and  column  of  the  matrix. 

To  use  the  constraint  matrix,  it  is  necessary  to  check  the  list  of  constraint  entries  found 
after  the  rule,  in  which  the  left  hand  member  is  the  first  subgoal  being  used  in  the  top-down 
strategy.  Thus,  one  rule  generally  lists  all  the  constraints  for  an  identification.  Each  time  a 
rule  is  used  in  the  identification  algorithm  the  list  of  constraint  entries  is  checked  to  see  if 
any  are  pertinent.  If  any  constraints  are  to  be  considered,  the  correct  constraint  is  obtained 
Irom  the  matrix  and  performed.  If  the  constituents  do  not  satisfy  the  constraint,  the  predicted 
identification  is  rejected  and  a  new  identification  is  begun  in  a  top-down  manner.  If  the 
constraint  is  satisfied,  the  present  identification  continues. 

An  example  using  the  grammar  rules  just  described  and  the  constraint  matrix  in  Table  2 
is  indicated  in  Figure  11.  This  applies  a  semantic  interpretation  to  the  syntactic  string  in  the 
form  of  the  label  <doghouse>.  The  complete  process  would  then  attempt  to  apply  the  remaining 
labels  to  the  syntactic  string.  In  Appendix  .9,2  is  contained  a  larger  grammar  for  pattern 
recognizing  two  dimensoinal  buildings.  The  constraint  matrices  are  also  included. 

All  possible  semantic  interpretations  are  obtained  for  the  figure.  Thus,  if  in  using  the 

top-down  technique  the  figure  is  identified,  the  process  does  not  stop  until  all  possible  identi¬ 

fication  are  tried.  If  the  figure  receives  a  singular  identification,  the  segment  of  the  syntactic 
string  of  the  scene  which  represents  this  figure  is  assigned  the  label  or  identification  which 

has  been  obtained.  If  an  additional  figure  is  contained  in  the  scene  another  segment  of  the 

syntactic  string  representing  a  second  figure  is  sent  to  the  first  level  grammar  of  the  semantic 
component.  Again  the  procedure  for  semantically  interpreting  the  figure  is  followed. 


The  actual 
as  follows: 
1. 

2. 

3. 


fable  2.  hxample  of  first  Level  Grammar  Constraint  Matrix 
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Figure  11.  Example  of  Semantic  Interpretati 
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When  all  figures  ol  the  scene  have  been  processed  by  the  first  level  grammar,  the  semantic 
interpretations  of  the  figures  are  sent  to  the  second  level  grammar. 

6.4  THE  SECOND  LEVEL  GRAMMAR  OF  THE  SEMANTIC  COMPONENT 

While  the  first  level  works  in  terms  of  the  syntax  of  the  figure  (i.e.,  the  context  of  the 
primitives)  the  second  level  works  in  terms  of  the  syntax  of  the  scene  (i.e,,  the  context  of 
the  figures).  Once  all  the  figures  of  the  scene  have  been  operated  upon  by  the  first  level 
grammar  and  the  resulting  semantic  interpretations  have  been  sen.  to  the  second  level  grammar 
one  of  the  following  situations  has  been  obtained  by  the  first  level  grammar  and  the  correspond¬ 
ing  subsequent  steps  indicated  are  taken. 

'•  Eae!l  fi8ure  of  the  scere  has  received  a  singular  semantic  interpretation. 

In  this  case  the  semantic  labels  are  operated  upon  by  the  rules  of  the 
second  level  to  obtain  a  semantic  label  for  the  scene. 

2.  Each  figure  of  the  scene  has  received  at  least  one  semantic  label  and  at 
least  one  figure  has  received  more  than  one  semantic  label,  that  is. 
the  figure  is  considered  ambiguous.  In  this  case  the  set  of  all  combin’ 
ations  of  labels  of  the  figures  is  processed  by  the  second  level  grammar. 

The  second  level  grammar,  whose  rules  define  the  context  of  the  figures 
may  eliminate  some  of  the  semantic  interpretations  which  cause  one  or 
more  of  the  figures  to  be  ambiguous.  It  is  possible  that  after  the  process¬ 
ing  by  the  second  level  grammar  the  figure  will  still  be  ambiguous.  In 
this  case  the  scene  will  receive  more  than  one  ijemantic  interpretation, 
making  it  ambiguous. 

3‘  At  kaSt  one  fi8Ure  cf  the  sce"e  has  received  at  least  one  semantic  inter¬ 
pretation  and  at  least  one  figure  has  received  no  semantic  interpretation, 
that  is,  the  figure  is  considered  anomalous.  In  this  case  a  deductive 

inference  procedure  will  be  used  to  attempt  to  semantically  interpret  the 
anomalous  figures. 

The  first  and  second  case  of  above  use  the  same  procedure  of  a  bottom-up  strategy  in 
applying  the  rules  of  the  second  level  grammar.  The  third  case  in  addition  uses  a  form  of 
deductive  inference  to  semantically  interpret  the  scene.  Its  procedure  is  outlined  in  Chapter  7 
under  the  section  on  future  extensions. 

The  rules  of  the  second  level  grammar  are  of  the  same  form  as  the  first  level  grammar 
that  is, 

X=Y,W,*Ri,Z,*Rj,.../0 

where  Y,  W,  and  Z  are  constituents  related  by  the  indicated  relations.  The  0  is  again  a  con¬ 
textual  constraint  and  has  the  same  definition  as  that  for  the  first  level  grammar  The  con¬ 
straints  are  placed  in  matrices  as  they  were  for  the  first  level  grammar.  An  example  of  such  a 
matrix  ,s  shown  in  Table  3,  where  house=I,  garage  =  2,  and  doghouse=3.  A  sample  of  the 
second  level  grammar  is  as  follows. 

<scene>:;=<baekyard>  |  Chome^ 

<home- : :=<house>  <backyard^  ^horizontal  relation>  | 

<house">  <garage>  Chorizontal  relation^  | 

<house>  <doghouse">  <horizontal  relation^-  /  (1 ,1),(1 ,2), 

(1, 3), (2, 2), (3, 3) 


Table  3.  Lxample  <<t  Second  Level  Grammar  Constraint  Mat'ix 
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backyard  > :  :=<garage'>  <doghou  se~>  <horizontal  relation'-/ (2, 2), (2, 3), 

(3,3) 

In  the  second  case,  where  there  is  the  possibility  of  the  scene  receiving  more  than  one 
label,  those  combinations  of  labels  of  the  figures  which  are  not  well-formed  according  to  the 
rules  of  the  second  level  grammar  are  rejected.  The  procedure  in  using  the  grammar  and  the 
constraint  matrices  is  the  same  as  that  described  for  the  first  level  grammar. 

The  third  case  requires  some  form  of  deductive  inference  to  supply  enough  information 
to  use  the  rules  of  the  second  level  grammar.  The  procedure  described  in  Section  7.1  is 
actually  developed  for  the  more  complex  situation  in  which  more  than  two  figures  make  up  a 
scene. 

If  the  figure  cannot  be  identified  by  either  the  first  or  second  level  of  the  semantic  com¬ 
ponent  grammar  then  the  figure  may  either  be  truly  anomalous  or  the  grammar  is  not  satisfactory 
for  the  graphic  data  being  considered.  An  unsatisfactory  grammar  may  be  due  to  the  fact  that 
the  grammar  is  incomplete  and  hence  just  not  extensive  enough  or  the  particular  grammar  being 
used  may  be  a  poor  choice  for  the  particular  graphic  data.  A  grammar  which  is  thought  not  to 
be  sufficiently  extensive  can  most  likely  be  corrected  without  an  unreasonable  amount  of 
difficulty.  If  the  grammar  is  considered  a  poor  choice  either  a  new  set  of  primitive  s  and/or 
syntactic  relations  is  needed  to  parse  the  data  or  a  new  set  of  grammar  rules  using  the  same 
primitives  ^nd  syntactic  relations  is  required. 


7.  FINAL  CONSIDERATIONS 


7.1  FURTHER  EXTENSIONS 

The  following  list  provides  some  of  the  many  extensions  which  can  be  considered  for 
SAP  and  for  syntax-directed  models  in  general, 

1.  The  ability  for  the  model  to  learn  is  a  natural  development  in  using  a 
grammar.  A  higher  level  of  learning  would  allow  the  model  to  have  some 
self-organizing  or  inductive  inference  capabilities4  3-4  4  in  order  to  extend 
its  grammar  as  it  sees  fit,  modifying  or  redefining  rules  of  the  grammar 
when  necessary. 

2.  The  use  of  deductive  inference  mentioned  in  section  6.4  is  not  as  ambi¬ 
tious  as  the  preceding  extension  and  thus  deserves  some  outlining.  The 
use  of  deductive  inference  can  be  applied  to  the  situation  in  which  at 
least  one  figure  of  a  scene  has  received  at  least  one  semantic  inter¬ 
pretation  and  at  least  one  figure  has  received  no  semantic  interpretation. 

As  mentioned  in  section  6.4,  the  procedure  is  actually  of  significant 
value  only  if  more  than  two  figures  comprise  a  scene.  The  inference  is 
made  assuming  an  identification  of  one  of  the  unlabeled  figures.  Using 
this  assumption,  identifications  are  obtained  for  various  other  figures 
and  these  identifications  are  checked  for  contradictions.  If  a  contra¬ 
diction  (rcductio  ad  absurdum)  does  arise,  the  original  assumption  is 
known  to  be  wrong  and  hence  that  particular  meaning  is  eliminated  as  a 
possible  identification  of  the  figure  to  which  it  was  assigned.  Using 
an  inference  making  procedure  necessitates  detailed  bookkeeping  so  that 
a  distinction  can  be  maintained  as  to  those  figures  which  have  been 
rigorously  identified  and  those  figures  which  have  been  identified  on 
the  basis  of  the  assumed  identification  of  another  figure.  If  an  assumed 
identification  is  found  to  be  incorrect  due  to  the  constraints,  any  identifi¬ 
cations  based  on  the  incorrect  assumption  must  be  removed  by  a  back¬ 
tracking  procedure, 

The  above  use  of  deductive  inference  can  be  applied  to  actually  perform 
either  of  two  types  of  identification.  They  are: 

(i)  K-nowing  that  a  particular  identified  figure  is  in  the  scene,  an 
unidentified  figure  is  completely  identified. 

(ii)  Knowing  two  partially  identified  figures  in  a  scene,  each  is 
identified  from  information  of  the  other. 

In  the  second  type  of  deductive  identification  situation,  the  partial 
identification  would  come  from  the  first  level  grammar  of  the  semantic 
component,  The  difficulty  is  in  extrapolating  these  partial  identifications 
to  deductive  inferences  by  the  second  level  grammar.  This  is  an  inter¬ 
esting  problem  which  has  yet  to  be  considered  in  the  literature, 

3.  The  possibility  of  merging  the  generation  and  pattern  recognition  modes 
into  one  totally  integrated  system  provides  for  further  learning  capabilities 
plus  a  high  degree  of  man/machine  interaction. 

4.  The  use  of  a  two-dimensional  grammar  in  man/machine  interaction  would 
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atlow  the  user  to  define  grammar  rules  without  needing  an  extensive 
knowledge  of  the  system.  A  recent  Ph.D.  Thesis  by  T.A.  Standish45 
luts  indicated  an  approach  to  this  problem. 

5-  I  he  use  of  syntactic  analysis  to  allow  dynamic  pictures  to  be  generated 
has  unlimited  applications. 

6.  Without  a  doubt,  the  most  sophisticated  extension  is  the  ability  to  operate 
with  figures  which  contain  information  concerning  depth.  The  c|uantum 
jump  from  two  to  three  dimensit  n  would  produce  problems  manyfold, 
but  it  pattern  recognition  is  to  be  of  any  truly  significant  value,  these 
problems  must  be  tackled  and  solved. 

7.2  ADVANTAGES  OF  THE  SYNTAX-DIRECTED  MODEL 

Ihe  advantages  of  the  general  syntax -directed  model  for  pattern  analysis  can  be  described 
briefly  by  the  following  points; 

1.  What  appears  to  be  the  strongest  point  of  syntax-directed  analysis  is  its 
ability  to  analyze  an  arbitrarily  complex  pattern.  The  ease  with  which 
recursiveness  can  be  placed  within  the  grammar  allows  an  infinite  number 
of  variations  in  the  input  patterns  to  be  identified. 

2.  The  ability  to  analyze  arbitrarily  complex  figures  indicates  a  high  degree 
of  abstractness  may  be  represented  by  the  grammar,  allowing  a  wide  range 
of  classes  to  be  defined  by  the  grammar. 

3-  Ihe  use  of  the  grammar  allows  detailed  distinctions  to  be  made  as  fine 
as  desired. 

4.  Ihe  use  of  a  grammar  which  decomposes  a  pattern  into  its  simpler  con¬ 
catenated  parts  provides  not  only  the  name  of  the  pattern  but  a  structural 
description  which  efficiently  represents  the  pattern. 

5.  The  descriptive  power  of  the  grammar  provides  the  ability  to  determine 
topological  equivalence  among  patterns  in  addition  to  providing  a  basis 
upon  which  the  semantic  analysis  can  be  made. 

6.  The  consideration  of  graphics  displays  which  have  dynamic  capabilities 
is  provided  an  interesting  approach  by  the  syntax-directed  analysis  of 
figures.  Because  the  topological  features  of  the  figure  are  available, 
it  can  be  readily  translated  by  translating  any  single  constituent  of  the 
figure. 

7.  The  use  of  the  topological  features  to  represent  a  figure  is  an  extremely 
efficient  manner  in  which  to  store  pictorial  data.  Rather  than  store  all  of 
the  digitized  points,  the  name  of  a  higher  level  constituent,  such  as 
triangle,  is  stored  along  with  its  reference  point.  This  reference  point 
and  the  name  of  the  constituent  completely  determine  the  set  of  points 
which  comprise  the  constituent. 

8.  Also  because  the  topological  features  are  available  and  hence,  the  con¬ 
stituents  of  the  figure  are  related,  additions  to  the  figure  may  be  made 
by  essentially  augmenting  the  structural  description  of  the  figure  without 
altering  the  already  existing  structural  description. 

9.  The  use  of  a  grammar  allows  the  syntax-directed  analysis  to  be  invariant 
under  linear  displacement  and  size. 
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10  Th e  grammar  is  easily  extendable  and  may  be  altered  with  no  ramifi¬ 

cations  in  most  cases. 

11.  The  efficacy  by  which  the  grammar  may  be  extended  provides  an  oppor¬ 
tunity  to  add  sophisticated  learning  techniques  to  the  syntax-directed 
models. 

12.  A  further  advantage  of  the  ease  by  which  the  grammar  may  be  changed 

allows  for  a  wider  range  of  patterns  to  be  recognized  without  large 

changes  in  the  syntactic  and  semantic  components. 

13.  A  strong  factor  which  has  just  begun  to  be  utilized  is  the  ability  of  the 

grammar  to  operate  as  either  a  generator  or  recognizer  of  patterns.  This 

allows  for  a  high  degree  of  man  machine  internet  on  without  a  great  deal 
of  overlapping  operators. 

14.  The  ability  of  the  syntax-directed  pattern  recognizer  to  operate  in  a 
parallel  mode  may  yet  prove  to  be  its  strongest  point.  Using  a  top-down 
strategy  various  alternatives  may  he  considered  simultaneously  by  using 
a  grammar.  This  should  prove  to  be  an  enormous  time  saving  factor  and 
also  serves  as  a  check  for  errors  in  the  identification  process. 

15.  Because  the  syntax-directed  analysis  processes  all  the  information  in 
an  organized  and  meaningful  way.  both  local  and  distant  Gestalt  qualities 
are  obtained.  This  is  particularly  appealing  to  psychologists  and  physio¬ 
logists  who  are  r  versing  the  normal  research  behavior  by  studying  the 
techniques  used  in  computer  programs  to  hypothesize  about  the  workings 
of  the  human  conceptual  and  physical  processes. 

.3  DISADVANTAGES  OF  THE  SYNTAX-DIRECTED  MODEL 

The  disadvantages  of  such  a  model  can  be  described  by  the  following  points: 

1.  Not  only  the  strength,  but  also  the  weakness  of  the  model  lies  in  the 
grammar.  The  choice  of  primitives,  syntactic  relations,  and  higher 
level  constituents  to  be  formed  essentially  distinguishes  one  grammar 
from  another.  This  choice  is  critical  and  for  syntactic  analysis  to  reach 
any  degree  of  sophistication,  a  formalization  lor  this  choice  must  be 
developed. 

2.  What  may  prove  to  be  a  major  weakness  of  syntactic  analysis  is  the 
multiple  parsings  which  are  obtainable  for  a  figure.  These  multiple 
parsings,  however,  do  not  appear  to  carry  the  same  degree  of  ambiguity 
that  they  do  in  natural  language  analysts  though  there  is  a  seemingly 
astronomical  number  of  different  parsings  that  can  be  obtained  from  a 
figure  of  only  reasonable  complexity.  If.  as  it  appears  likely,  all  of 
these  parsings  would  receive  the  same  semantic  interpretation,  then 
rather  than  process  all  of  them,  the  grammar  needs  to  be  developed  to 
be  able  to  semantically  interpret  any  one  of  these  parsings.  As  this 
solution  is  infeasible,  the  syntactic  analysis  needs  to  obtain  the  parsing 
in  a  normalized  manner  and  the  grammar  which  provides  the  semantic 
interpretation  is  then  designed  to  expect  the  parsing  in  this  normalized 
form.  If  the  multiple  parsings  of  a  figure  do  receive  different  semantic 
interpretations  then  it  is  necessary  to  obtain  all  of  them.  In  this  case. 
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to  eliminate  some  of  the  parsings  it  is  necessary  to  define  higher  level 
primitives  which  absorb  structural  attributes  not  significant  in  the  pattern 
recognition  process.  (This  is  done  in  SAP  by  defining  triangles,  etc. 
to  be  higher  level  primitives  upon  which  the  semantic  component  oper¬ 
ates.) 

3.  The  problem  of  multiple  parsings  can  be  considered  to  create  a  semantic 
analysis  problem.  A  second  semantic  analysis  problem  is  the  use  of  the 
structural  descriptions  by  the  semantic  component  grammar.  The 
process  is  considered  slow  and  not  efficient  because  of  the  number  of 
false  starts  by  either  the  top-down  or  bottom-up  techniques.  Parallel 
processing  should  be  a  solution  to  this.  An  alternative  to  the  use  of  a 
grammar  for  the  semantic  analysis  is  the  use  of  discrimination  nets 
(decision  trees).  EPAM44,47  and  other  models  of  this  type48-49  have 
shown  the  discrimination  net  to  be  an  effective  decision  maker  and 
adaptable  to  various  learning  and  self-organizing  algorithms.  While  a 
net  does  not  represent  recursion  as  well  as  a  grammar,  it  has  not  been 
found  that  the  semantic  component  needs  a  highly  recursive  mechanism. 

4.  It  is  possible  that  a  grammar  will  not  be  able  to  resolve  ambiguities  as 
some  ad  hoc  pattern  recognition  techniques.  One  difficulty  which  has 
been  resolved  but  continues  to  cause  some  anxiety  is  the  measuring  of 
distances  between  edges.  The  following  two  examples  should  indicate 
the  problem.  (i)  In  the  drawing  below,  the  distance  of  a  horizontal 
relation  between  primitive  R1  and  the  constituent  formed  from  primitives 
R2  and  R3  is  shown.  A  user  might  be  tempted  to  consider  the  distance 
to  be  that  between  Rl  and  the  left  edge  of  R2. 


KV 

(ii)  A  possible  difficulty  in  syntactic  representation  may  also  be  seen 
by  the  example 

R(  1 ,4,8),  R(2,8,4),  *R4(TOP),  R(3.8,4),  *R1(LE4LT) 


give  s 


-8 ,- 


while 

R ( 1 ,4.8),  R(2,8,4),  *R4(T0P),  R(3.8,4),  * R 1  ( L E 3 L T > 


5.  Because  most  complex  graphical  data  has  contextual  constraints,  the 
responsibility  on  formulating  these  constraints  to  avoid  mislabeling 
ambiguous  or  anomalous  figures  lies  on  the  person  creating  the  grammar. 

6.  The  choice  as  to  the  primitives  of  the  grammar  is  made  difficult  by 
fact  that  the  primitives  are  essentially  contained  implicitly  in  the 
graphic  data. 

7.  The  division  between  hardware  and  software  is  a  point  of  detentior  in 

syntactic  analysis.  Syntactic  analysis  is  essentially  a  software  tool 
while  several  aspects  of  the  analysis  may  be  performed  better  by  a 
hardware  component.  An  example  is  the  syntactic  description  of  a 

circle,  which  should  perhaps  be  incorporated  in  the  hardware. 

7.4  COMPARISON  SUMMARY 


The  above  list  of  advantages  and  disadvantages  of  the  syntax-directed  model  for  pattern 
recognition  should  not  be  considered  independent  of  the  alternative  methods  for  pattern  recog¬ 
nition.  Using  Minsky's50  classification  of  pattern  recognition  models  as  templet  matching, 
property  list  matching,  and  ungulate  descriptions  (syntactic  analysis),  the  advantages  and 
disadvantages  may  be  summarized  as  follows: 

I.  Numbers  1,2,4,5,6,8.11,13,15  of  the  list  of  advantages  of  the  syntax- 
directed  model  are  advantages  over  the  templet  and  property  list  matching 
techniques. 

11.  Numbers  3,7,9,10,12  of  the  list  of  advantages  are  advantages  over  only 
the  templet  matching  technique. 

HI.  Numbers  1,2, 5, 6  of  the  list  of  disadvantages  are  true  disadvantages 
compared  to  the  templet  and  property  matching  techniques. 
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8.  SUMMARY  AND  CONCLUSION 


The  model  SAP  presented  in  this  report  is  a  study  in  the  methodology  of  the  syntax 
directed  pattern  recognition  and  generation  of  pictures.  SAP  performs  a  many-to-one  mapping 
b>  accepting  pictorial  data  as  input  and  providing  as  output  a  label  for  the  data.  The  pictorial 
data  considered  are  two-dimensional  pictures  which  contain  no  depth  information. 

In  order  to  operate  as  either  a  recognizer  or  generator  of  pictures,  SAP  is  composed  of 
two  components,  a  syntactic  component  and  a  semantic  component.  In  terms  of  pattern  recog¬ 
nition.  the  syntactic  component  accepts  a  picture  1..  and  translates  it  to  a  set  of  strings  of 
symbols  L*.  which  describes  the  picture's  structure.  Thus,  the  function  of  the  syntactic 
component  is  translating  from  a  two-dimensional  language  L  to  a  one-dimensional  language  L*. 
lire  syntactic  component  uses  a  lexicon,  a  modified  phrase  structure  grammar,  and  a  set  of 
transformational  rules  to  perform  this  translation, 

I  he  semantic  component  accepts  structural  description  L*(.  from  the  set  of  structural 
descriptions  L*(  of  picture  L|  and  attempts  to  apply  a  semantic  interpretation,  designated  as 
Label...  to  the  picture.  First  a  set  of  abstractions  of  1..  are  obtained  by  various  operations 
on  the  string  ot  symbols  L*f..  to  obtain  a  general  classification  of  1...  The  string  L*..  is  then 
operated  upon  by  a  context  sensitive  modified  phrase  structure  grammar  to  receive  a  semantic 
interpretation.  Ihe  recognition  process  is  made  in  terms  of  the  syntax  and  context  of  the 
figures  which  comprise  the  picture. 

SAP  also  has  the  facility  to  allow-  a  high  degree  of  man  machine  interaction.  A  formal¬ 
ization  ot  the  syntax  L*  allows  a  user  to  by-pass  the  syntactic  component  and  have  the 
semantic  component  attempt  to  semantically  interpret  a  structural  description  which  he  has 
created  or  bv-pass  the  semantic  component  to  generate  a  picture  from  a  structural  description 
which  he  has  created. 

Aspects  ot  the  syntactic  component  of  SAP  which  appear  particularly  promising  are  the 
use  of  transformation  rules  to  add  artificial  lines  to  a  picture  and  the  combination  of  a  lexicon 
and  a  set  of  3-tuples  to  explicitly  represent  the  implicit  syntactic  relations  which  form  the  topo¬ 
logical  features  of  a  figure.  The  language  L*.  though  highly  restricted,  is  capable  of  represent¬ 
ing  the  structural  description  of  rather  complex  two-dimensional  figures,  which  in  turn  allows 
the  user  to  readily  interact  with  SAP, 

Similarly,  significant  contributions  of  the  semantic  component  of  SAP  are  the  use  of  an 
abstracting  process  to  obtain  a  general  classification  of  the  figure  and  the  ability  to  represent 
contextual  constraints  which  exist  between  various  levels  of  the  constituents  which  comprise 
a  figure  or  scene  (the  entire  picture).  These  tools  are  deviations  from  the  use  of  a  phrase 
structure  grammar  to  obtain  the  structural  description  and  semantic  interpretations  of  a  figure 
or  scene.  This  deviation  is  the  type  of  direction  that  must  be  taken  in  order  to  be  able  to 
syntactically  analyze  a  complex  two-dimensional  picture. 

In  conclusion,  the  ability  of  the  syntax-directed  model  to  be  able  to  recognize  or  generate 
an  arbitrarily  complex  figure,  determine  the  topological  equivalence  among  patterns,  readily 
apply  to  dynamic  displays,  proxide  an  efficient  approach  to  storing  a  pictorial  data  base,  and 
operate  in  a  parallel  mode  indicates  that  such  a  model  is  a  highly  desirable  method  of  operation 
for  dealing  with  at  least  some  types  of  pictorial  data.  Because  the  syntax-directed  model  does 
require  a  degree  of  formalization  of  the  structure  of  the  expected  pictorial  data  and  there  is 
difficulty  in  dealing  with  multiple  structural  descriptions  of  a  single  figure,  such  a  model  may 
not  be  necessarily  applicable  to  all  cases  of  pattern  recognition  and  generation.  However. 
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the  ability  to  extend  the  syntax-directed  model  to  perform  inductive  and  deductive  reasoning 
and  ultimately  to  process  three-dimensional  information  indicates  the  high  degree  of  sophis¬ 
tication  which  the  syn  tax  -directed  model  should  be  expected  to  achieve. 
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APPENDIX  9.1 

THE  IMPLEMENTATION  OF  SAP 

Various  aspects  of  SAP  have  been  implemented  on  the  LINC,  the  IBM  360  50,  and  the 
IBM  7072.  The  first  implementation,  Program  I,  was  on  the  LINC,  a  small  genera  1  p urpo se 
digital  computer,  to  study  the  type  of  pictorial  data  chosen.  The  programming  language  used 
is  an  assembly  language,  LAP6, 

Basically,  Program  l  is  composed  of  subroutines  which  aie  able  to  display  any  of  the 
geometric  primitives  for  any  specified  dimensions.  The  location  of  the  primitives  on  the  screen 
is  determined  by  giving  a  particular  value  to  their  reference  point.  The  information  is  entered 
through  the  keyboard,  and  thus  by  entering  a  group  of  referenced  primitives,  any  desired  scene 
may  be  displayed  on  the  storage  scope.  The  actual  notation  of  the  referenced  primitives 
is  displayed  on  the  LINC  screen,  allowing  changes  and  deletions  to  be  made  as  it  is  typed 
into  the  LINC.  The  photographs  in  Figures  9.1.1(a),  9.1.2(a),  and  9.1.3(a)  are  examples  of  the 
graphic  feedback.  The  horizon  is  obtained  by  referencing  a  rectangle  of  zero  height.  Also, 
the  circles  and  ellipses  are  generated  using  sine  and  cosine  tables,  rather  than  approximated 
by  eight  directional  lines,  as  described  in  Chapter  5. 

Program  II,  also  on  the  L.1NC  in  LAP6,  is  a  first  implementation  of  L**.  Thus,  after  a 
syntactic  string,  L*,.  is  typed  into  the  keyboard,  the  graphic  representation  of  the  string  is 
displayed  on  the  storage  scope.  Using  a  single  referenced  primitive  in  the  string,  the  reference 
points  of  the  other  primitives  are  obtained  by  the  syntactic  relations  relating  the  primitives. 
This  allows  the  user  to  see  a  display  of  the  parsing  he  has  just  typed  into  the  LINC.  With 
only  slight  extension,  this  same  procedure  will  provide  information  in  the  form  of  graphic 
displays  of  the  various  abstractions  performed  on  the  string  by  the  semantic  component. 

I  he  various  options  ot  the  graphic  syntactic  language  indicated  in  Chapter  4  are  not 
allowed  at  the  present  time.  While  more  efficient  compiling  routines  could  be  written  to  code 
the  input  string,  this  topic  is  incidental  to  the  development  of  SAP  and  has  been  ignored. 

In  Figures  9.1.  Kb/,  9.1.2(b),  and  9.1.3(b)  are  examples  of  the  input  strings  required  to  produce 
the  scenes  shown  in  Figures  9.1 . 1(a), 9.1 .2(a),  and  9.1.3(a).  It  should  be  noted  that  the  present 
implementation  cannot  yet  fully  generate  these  graphic  scenes  from  their  respective  syntactic 
string.  The  graphic  scenes  were  produced  by  the  referencing  technique  of  Program  I. 

A  program  on  the  IBM  360/50  has  been  written  in  LISP  1.5  to  perform  various  manipu¬ 
lations  on  the  triplet  set  and  transfoim  the  triplet  set  into  a  Reverse  Polish  string.  The 
algorithm  which  obtains  the  Reverse  Polish  string  from  the  triplet  set  bypasses  an  explicit 
formation  of  the  syntax  tree. 

In  Figure  9.1.4  is  a  flow'  diagram  of  the  program  which  obtains  the  Reverse  Polish  string 
from  the  triplet  set.  Section  9.1.1  contains  a  listing  of  the  program. 

The  entire  abstraction  subcomponent  has  been  programmed  in  FORTRAN  on  the  IBM  707  2. 
As  indicated  in  Chapter  6,  the  program  accepts  as  input  a  string  which  provides  a  structural 
description  of  a  scene.  The  abstraction  subcomponent  segments  the  string  into  individual 
tigures  and  abstracts  each  tigure.  placing  the  figure  into  one  ot  the  seven  general  classes 
listed  on  page  92.  These  classes  are  described  in  greater  detail  in  Chapter  6  in  the  discussion 
of  the  abstracting  component. 


ack  of  a 
e  9.1. l(a 


Scene 


TR(1 .10,15),  Rd.10,15),  *R1(LE),  R<2.2,1), 

R (3 , 1 . 5) ,  *R4,  R( 4 , 2 , 1 ) ,  *R4.  TKl, 30,20), 
TK2.7.10),  *R5(B0T),*R1,  R(5,4,12),  R(6,4,12), 
*R4,  R(7 , 7 , 1 5) ,  *R  1(13),  R(8,30,40),  *R5(B0T), 
*Rl,  *R4(B0T),  TL-(  1,10,1 5),  R(9,10,15),  *R1(LE), 
*R4(B0T),  E(l,  17,5),  *R3(TOP12UP),  E(2, 20, 5), 
*R4(10,TOP1DN),  E ( 3 ,7 , 30) ,  R(10,2,4),  *R1, 
*R4(2,BOT),  E(4,7 ,30),  R(ll,2,4),  *Rl,  *R4(B0T), 
R(0, 777, 0,0, -350),  *R1(RE8LT) 


The  Syntactic  String  to  Generate  the  Church  Scene 
Figure  9,1. 1(b) 
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i 


The  Syntactic  String  to  Generate  the  harm  Scene 
Figure  9.1.2(b) 


The  Graphic  Feedback  of  a  House  Scene 
Figure  9.1.3(a) 


C<  1,32,32),  R(l,3,10),  *R1,  Tl(l ,25,6), 
R(2,3,5),  R(3,4, 1 2),  *R4(l,BOT3UP), 
R(4,3,5),  *R3(1  ,B0T3DN),  (5,23,20), 
*R5(BOT),  *R1 ,  * R3(  1 5, BOT), 

F(  1 ,20,7),  *R1(7,RT10LT), 

R(0, 777, 0,0,-200),  *Rl(RT2LT) 


The  Syntactic  String  to  Generate  the  House  Scene 
Figure  9.1.3(b) 
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l-'igurc  9.1.4  A  blow  Diagram  of  the  I’rojtram  which  Obtains  a  Kc  verse  l’olisli  String  I  roin  the  lnplei  Set 


f  Input:  Structura 
Description  ( Rev  el  se 

I’olislt  St ri n y > 


K I '  String 
Coded  into 
Arms  "IADS  I 


f  Is  IADS  l  a 


Csmg  Rottom-Lp 
Right-to-I.elt  Strategv 
Abstractions  are  made 
Dv  t  esting  for: 
Constituent.  Constituent 
Sv ntactie  Relation 


{etwee 


f  Does  Relation 
f  Inch c at e  Noneont igmty 
i  Detween  Constituents'' 


r  Are 

( 'oust  ituent  s 
S.  Labeled"  . 


Does  Relation 
Indicate  Contained  Within 
l  Relationship7 


Abstract  Constituents 
Into  Higher  I .evel 
Constituent 


Label 

Constituents 


igurc  *>.1.5  A  Mow  Diagram  of  the  Program  Which  Segments  and  Ahstiacts  a  1'igiin 
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1.  STORE  5.  BULBOUS 

2.  MOUSE  6.  OFFICE  BLDG 

3.  SILO  7.  UNICORN 

4.  TREE 

The  structural  descriptions  can  be  of  any  complexity,  having  both  directional  recursion 
or  contained  within  recursion.  Figure  9,1.5  contains  a  flow  diagram  of  the  abstracting  program. 
Section  9,1.2  is  a  listing  of  the  program, 

9,1.1  A  Listing  of  the  Program  Which  Obtains  a  Reverse  Polish  String  from  the  Triplet  Set 


DEFINE  (( 

(RE  POL(LAMRDA(MAIN) 

(PROG(RPS  STATE  ARC.  X  REAM) 

(SETQ  X  (CAR  MAIN)) 

U  (CONIKNULL  ARC!)  (SETQ  ARC.(CONS(CAR  X)  NIL))) 

(TISETQ  ARC.  (CONS  (  CAR  X)  (CDR)ARG))))) 

V  (SETQ  ARC.  (CONS  (CADR  X)  ARG)) 

(CONDUNULL  RPS)  (SETQ  RPS  ( CONS(CADDR  X)NIL))) 

(T(SETQ  RPS  (CONS(CADDR  X)  RPS)))) 

<COND((NUl  L  STATE  )  (SETQ  ST ATE(CONS  1  NIL))) 

(T(SETQ  STATE  (CONS  1  STATE)))) 

(SETQ  MAIN  (EE  FACE  X  MAIN)) 

A  (CON D((  NULL  MAIN)  (GO  E))  (T  (SETQ  REAM  MAIN))) 

B  (SETQ  X  (CAR  REAM)) 

(COND((EQ(CAR  ARC.)(CAR  XD  (GO  V)) 

((EQ(CAR  ARG)  (CADR  X))  (GO  U))(T(SETQ  REAM(CDR  REAM)))) 
(COND((NULL  REAM)  (C.0  E))(T(C.O  B))) 

E  (SETQ  RPS  (CONS(CAR  ARG)  RPS)) 

(SETQ  ARG  (CDR  ARG)) 

F  (SETQ  STATE  (CONS(ADD!(CAR  ST ATE))(CDR  STATE))) 

(CONDOEQUAL  (CAR  ST ATE)3)(SETQ  STATEICDR  STATE))) 

(T(GO  A))) 

(COND((NOT(NULL  STATF.))(GO  F))(T(PRINT  RPS)))))) 
(EFFACE(LAMBDA(XX  LL) 

(CONDMNULL  LL)  NIL)((EQUAL  XX  (CAR  LL)) 

(CDR  LL))  (T(RPLACD  (LL(EEFACE  XX  (CDR  LL)))))))))) 


9.1.2  A  Listing  of  the  Program  Which  Segments  and  Abstracts  a  Figure 


DIMENSION  INPUT  (400LNDATA  (50,4) 

DIMENSION  MIN AB(  10) ,ISS(1 ,3),I ABST(2()(),3),MAXA B(  10) 
DIMENSION  IAB(3) 

DIMENSION  IIIOLD(3,3),  ITITE  U6) 

£’ ************************************************************ 
C  COMPILE  THE  STRUCTURAL  DESCRIPTION  STRING  TO 

C  CODED  TABULAR  FORM 

(’************************************************************ 
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DO  8000  IJKL  1,10 
DO  41  J  1,3 
DO  41  I  1,200 
41  lABST(IJ)  0 

READ  (1,9049)  ITITli 

9049  FORMA  T  ( 1 6 A 5) 

WRITE  (2,9049)  ITITE 
READ  (1,9000)  N 

9000  FORMAT  (11) 

NN  N*80 

RFAD(  1,9001)  (INPUT(I),  1  l.NN) 

9001  FORMA  T(80  A 1) 

WR1TF(2,90  50) 

9050  FORMA  R25IIO  STRUCTURAL  DESCRIPTION) 
WRIT  12(2 ,9002)  (INPIJT(I),  I  l.NN) 

9002  FORMA T(  HI  115A1) 


DO  5  1  l.NN 

5  INPUT(I)  XABSFdNPUT(D) 


*  *  **  ^ 


C  STRING  WILL  BH  PUT  IN  ARRAY  NDATA  WITH 
C  CONSTITUF.NTS  COD  HD 

C  TH STING  FOR  THE  TYPE  CONSTITUENT  WHERE  ALPHANUMERIC 
C  EQUIVALENTS  ARE  79  R,  83  T.  63  C.65  =  E.  26  * 


J  0 
K  0 

9  .1  .1  +  1 

10  IFONPUTU)  -  7900000000)  20,100.20 
20  IF(II1PI)T(J)  —  8300000000)  30.200.30 
30  IFONPUTU  )-  6300000000)  40.300,40 
40  lE(INPUTU)  -  6500000000)  50,400,50 
50  IF(INPUT(J)  -  2600000000)  8000,500.8000 


C  RECTANGLE  CODED  AS  9 


(•***  +  *  K 


100  K  K  t  1 
L  1 

NDATA  (K.L)  -  9 


(•**  > 
C 
C 


TESTING  FOR  DELIMITERS  WHERE. 
36  (.  16  ),  35  ,,  90  0.  91  L... 


^- *********** 


101  ,1  ,1  t  1 

102  IFUNPUT(J)  —  3  6000  00000)  103,101,103 

103  IFONPUTU)-  1600000000)  104,150.104 

104  IFONPUTU)  -  3500000000)  105.101,105 

105  IFONPUTU)- 9000000000)  8000.106,106 


-94- 


106  1ST  AT  =  0 


C  CODING  NUMBERS  TO  INTEGER  FORM 
£'************************************************************ 
107  NUM  =  (INPUT(J)  100000000-90 


J  =  J  - 1 

IF(INPUT(J)  -  9000000000)  1  10,108,108 

108  NUM  =  NUM*  10  i  ((INPUT! J)  1000000000)-  90) 

109  J  J  -  1 

110  L  L  -  1 
NDATA(K.L)  NUM 

I  FUST  AT)  102.102,510 

C  TESTING  FOR  TYPE  OF  TRIANGLE  WHERE, 

C  69  I,  79  =  R,  73  =  L ,  64  D 

£•************************************************************ 


200  .1  =  J  *  1 


IF(INPUT)J)  -  6900000000)  210.260,210 
210  IF(INPUT)J)  -  7900000000)  220,235,220 
220  1F(INPUT(J)  -  7300000000)  8000,248,8000 
(^’ ***************************************** 
C  TR  CODED  AS  11 
C  TRD  CODED  AS  12 


235  J  .1*1 
K  K  -  1 
L  1 

I E( IN PUT(J)  -  6400000000)  237,236,237 

236  NDATA(K.L)  12 
GO  TO  101 

237  NDATA(K.E)  11 
’GO  TO  102 


(_*  *  *  M  *  jf  ****************  * 

C  I  L  COD  hi)  AS  13 
C  TL1)  (  01)1:1)  AS  14 


.•****  *  *****  * 


•  *  *  *  * 

***********> 

************* 

******************************** 

248 

.1  .1*1 

K  k  •  1 

1  .  1 

11  (IN  IM  1  (\) 

-  6400000000) 

250,249,250 

249 

NI)A  1  A(K  I.) 

14 

GO  1  O  101 

250 

NDA  I  A(K  1.) 

1  3 

GO  1  O  102 

.i)5- 


C  ISOSCELES  TRIANGLE  CODED  AS  10 

^***********>i 

260  K  =  K  +  1 
L=  I 

NDATA(K.L)  =  10 
GO  TO  101 


******************** 


300  K  Kfl 
L  =  1 

NDATA(K.L)  7 
GO  TO  101 

|c  *  *  *  ******  * 

ELLIPSE  CODED  AS  8 

fc*******5! 

400  K  K  *  1 
L  =  1 

NDATA(K.L)  8 
GO  TO  101 


C  TESTING  FOR  SYNTACTIC  RELATIONS 

C  RELATIONS  ARE  CODED  AS  THEIR  REPRESENTATIVE  INTEGER 


150  J  -  I  t-  I 

1F(IN PL)T(J)  -  3500000000)  800,9.800 
500  J  J  +  1 

IFdNPUT(J)  -7900000000)  8000,505,8000 
505  J  =  J  t  I 

IF(INPUT(J)  -9000000000)  8000,8000,506 


506  K  =  K  t  1 
L  1 

NDATA(K.L)  (INPUT(J)/100000000)  -  90 

.!  =■  .1  t  I 

IFdNPlJT(J)  -  3600000000)  8000,507,8000 

507  .1  J  t  I 

IFdNPl)T(J)  -  9000000000)  8000,508,508 

508  ISTAT  I 
GO  TO  107 

510  I F'(IN PUT(J)  -  1600000000)  511,150,51  1 

51 1  J  J  t  I 

GO  TO  510 


C  CALCULATING  AREA  OF  PRIMITIVES  AND  STORING  THE 
C  TABLE  IN  ARRAY  1ABST 
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800  I  1 

MINAB(l)  =  1 
MAX  =  K 

DO  808  K  =  1,  MAX 
IA  BST(K ,  1)  -  N  DAT  A(K ,  1) 

1ABST(K  ,2)  =  N  DATA(K  ,2) 

IFTNDAT A(K ,  I)  -  7)  808,803,804 

803  RAD  =  NDAT A(K ,3)/2 
1ABST(K,3)  =  3,14*RAD**2 
GO  TO  808 

804  IF(NDATA(K,l)-  9)  803,805,807 

805  1ABST(K,3)  =  NDAT  A(K  ,3)*N  DAT  A(K  ,4) 

GO  TO  808 

807  IABST(K,3)  =  NDAT  A(K  ,3)*N  DAT  A(K,4)/2 

808  CONTINUE 

£************************************************************ 

C  TABULAR  FORM  OF  STRING 
£************************************************************ 
WRITE(2, 9060) 

9060  FORMAT142H0  TABULAR  FORM  OF  STRUCTURAL  DESCRIPTION) 
WRITE12,  9061) 

9061  FORMAT147H  PRIMITIVE  TYPE  NUMBER  A 

XREA) 

WRITE(2,666)((IABST(NI,NJ),NJ  =  1,3),NI=  1,K) 

666  FORMATUH  3115) 

(-•I************************************************************ 

C  THE  REMAINDER  OF  THE  PROGRAM  ABSTRACTS  AND  LABELS 
C  THE  FIGURE  WHILE  EACH  FIGURE  IS  SEGMENTED  FROM 
C  THE  ENTIRE  SYNTACTIC  STRING 
^ *****************************************************  ******* 

MAX  AB(  l)  =  K 

^' ************************************************************ 

C  TESTING  FOR  RELATION  INDICATING  NONCONTIGUITY 
£ ************************************************************ 

813  IF(IABST(K,1) -4)  809,809,810 

809  IF(IABST(K, 2))  821,821,812 

810  I  F(IA  BST(K  ,  1) -6)  811,811,900 
812  ITIT=1 

692  K  K—  1 

IF(K-MINABO))  8000,690,690 

(-**** ************************************************  ******** 

C  COMBINING  TWO  INDENTIFIED  FIGURES  WHERE  THE 

C  IDENTIFIED  FIGURES  ARE  LABELED  20 

£■*  *****************************************************  ****** 

690  I F(I ABST1K  ,  I)  —  20)  814,691,814 

691  ITIT  =  ITIT  f  1 

IFUTIT  -  2)  692,692,693  “ 
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693  ISS(1, 1)  =  20 
GO  TO  694 

£****  ************************************  ******************** 

C  TESTING  I  OR  SSS,SSS, SYNTACTIC  RELATION 
£***************************!<******************************** 

821  IT  =  1 

815  K  =  K-  1 

818  IF(K  -  MINAB(D)  8000,816,816 

816  IF(IABST(K,  1)  —  6)  817,817,825 

817  IF(IABST(K,  1)  —  5)  814,811,811 
811  K  =  K  -  1 

IF(IABST(K,  1)  —  6)  817,817,820 
820  IF(IABST(K,  1)  —  15)  823,900,823 
823  K  =  K  -  1 

IF(IABST(K,  1)  —  15)  819,900,693 

819  IF(IABST(K,  1)  —  6)  817,817,900 

825  IF(IT  1)  900,826,850 

826  IT  =  2 

GO  TO  815 

£*******jMc******>Jt******************************************** 

C  ABSTRACTING  TWO  CONSTITUENTS  TO  A  NEW  CONSTITUENT 
C  LABELED  15 

C  ARRAY  ISS  CONTAINS  NEW  CONSTITUENT 

£************************************************************ 

850  ISS(1 , 1)  =  1 5 

694  JF(IABST(K,3)  —  IABST(K  +  1,3)  855,855,857 
855  IF(IABST(K  +  1,1)-  15)  852,85  1,851 

851  ISS(1 ,2)=  IABS  T(K  +  1,2) 

GO  TO  861 

852  ISSd ,2)  =  I ABST(K  +  1,1) 

GO  TO  861 

857  IF(IABST(K, I)— 15)  854,853,853 

853  !SS(1,2)  =  IABST(K,2) 

GO  TO  861 

854  ISS(1, 2)  =  IABST(K,  1) 

861  IF(1ABST(K  +  2, 1)  —  5)  858,863,863 

858  ISS(1 ,3)  =  IABST(K,3)  +  IABST(K  +  1,3) 

GO  TO  864 

863  ISSd, 3)  =  IABST(K  f  1,3) 

£************************************************************ 

C  ARRAY  IHOLD  CONTAINS  CONSTI1  UENTS  WHICH  WERE 
C  USED  IN  THE  MOST  RECENT  ABSTRACTION 
£************************************************************ 

864  IM  =  0 
KOOK  =  K  +  2 

DO  867  UK  =  K,  KOOK 
IM  =  IM  +  1 
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DO  867  JJ  =  1,3 

867  1H0LD(1M,JJ)  =  1ABST(11K,JJ) 

£***  ********************************************************* 

C  CONSTITUENTS  USED  IN  THE  ABSTRACTION  ARE  ZEROED  OUT 
C  AND  REMOVED  FROM  THE  NEW  TABLE 

C *********************************************** ************* 

IABST(K.l)  =  0 
IABSTIK  +  1 ,1)  =  0 
DO  856  J  =  1,3 

856  IABST(K  +  2,J)  =  ISS(1,J) 

7098  ISTAR  =  MINAB(I) 

IFIN  =  MAXAB(l) 

1  =  1+  1 

MlNAB(l)  =  1 F  IN  +  1 
NOM  =  1 F  IN 

DO  875  J  =  1STAR.1F1N 
1F(1ABST(J,1))  8000,875,859 

859  NOM  =  NOM  +  1 
DO  860  J  J  =  1,3 

860  1ABST(N0M,J  J)  =  IABSTIJ  ,J  J) 

875  CONTINUE 

c************************************************************ 

C  WRITE  CUT  OF  RESULTS  OF  LAST  ABSTRACTION 

c************************************************************ 

MM  =  MIN  ABO) 

1ABNO  =  1—1 
WR1TE(2,  8011)  1ABNO 

8011  FORMAT(22HO  ABSTRACTION  NUMBER, 15) 

WRITE(2,880)((IABST(M,N),N  =  1,3),  M  =  MM,  NOM) 

880  FORMAT  (1H  3115) 

MAXAB(I)  =  NOM 
IT  =  0 

GO  TO  813 

c* ************************************************** ********* 

C  FIGURES  ARE  CLASSIFIED  BY  THEIR  ABSTRACTED  STRINGS 
C  ACCORDING  TO  THE  RELATIVE  AREA  AND  RELATIONS 
C  BETWEEN  THE  PRIMITIVES 

Q **********  ************************************************** 

900  IF(IABST(K,1)  —20)  899,8000,899 
899  IF(IABST(K,1)-15)  927,905,927 
927  IF(IABST(K,1)  —  9)  904,974,904 

904  IF(IABST(K,1)-  10)  8989,936,8989 

905  IFlIHOLDO ,1)  -  1 5)  944,945,945 

944  IAB1  =  IHOLD(l ,  1) 

GO  TO  906 

945  IAB1  =  IHOLD(l,2) 

906  IF(IHOLD(2,l)  -  15)  947,948,948 
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947  IAB2  =  IHOL  D(2 . 1 ) 

GO  TO  970 

948  IAB2  =  IHOLD(2,2) 

970  IFOAB 1-9)  901,915,90  1 

901  IFOAB1  -  7)  902,920,902 

902  IFUAB1  -  8)  903.920.903 

903  IF(IABl-lO)  8989,925,8989 

915  IFOAB2  -  9)  8989,9  16.8989 

916  lF(IHOLD(3,l)-  5)  976,975.976 

974  !ABST(K,2)  lABST(K.l) 

975  WRITE  (2,977) 

977  FORMATU6H  FIGURE  IS  STORI 
GO  TO  999 

976  WRITE(2,917) 

917  FORMAT(23H  FIGURE  IS  OFFICE  BLDG.) 

GO  TO  999 

920  IFUAB2  -9)  8989,921,8989 

921  WRITE(2,922) 

922  FORMAT(18H  FIGURE  IS  BULBOUS) 

GO  TO  999 

925  I F (I A B 2  —9)  8989,926,8989 

926  IF(IHOLD(3, 1)  -  I)  8989,930,8989 

930  IF(3*IHOLD(l,3)  -  1H0LD(2.3))  940,931,931 

931  IF(IHOLD(l,3)  -  2*IHOLD(2,3))  933,936,936 
933  WRITE  (2,935) 

935  FORMAT  (16H  FIGURE  IS  HOUSE) 

GO  TO  999 

936  WRITE  (2,937) 

937  FORMAT  (15H  FIGURE  IS  TREE) 

GO  TO  999 

940  WRITE  (2,941) 

941  FORMAT  (12H  FIGURE  IS  SILO  TYPE) 

GO  TO  999 

8989  WRITE  (2,8990) 

8990  FORMAT  (18H  FIGURE  IS  UNICORN) 
£************************************************************ 
C  A  CLASSIFIED  FIGURE  IS  CODED  20 

^  ************************************************************ 


999  IABST(K ,  I)  =  20 
K  =  MAXAB  (I) 
GO  TO  813 
8000  CONTINUE 
END 
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APPENDIX  9.2 

A  LISTING  OF  THE  RULES  OF  THE  SYNTACTIC  AND  SEMANTIC  COMPONENTS 

The  following  pages  contain  a  listing  of  the  rules  which  are  used  in  SAP.  The  rules 
are  presented  in  the  order  listed  below. 

1.  The  Syntactic  Component  Grammar  Kl*. 

2.  The  1  ranslormation  Rules  for  Forming  Artificial  Lines. 

3.  The  Transformation  Rules  for  the  Concatenation  of  Lines. 

4.  The  Lexicon. 

5.  The  Transformation  Rules  for  the  Triplet  Set. 

6.  The  Grammar  K2*  for  L*. 

7.  The  hirst  Level  of  the  Semantic  Component  Grammar  K3*. 

8.  The  Constraint  Matrices  for  First  Level  of  K3*. 

9.  The  Second  Level  of  the  Semantic  Component  Grammar  K3*. 

10.  The  Constraint  Matrices  for  the  Second  Level  of  K3*. 

1.  Syntactic  Component  Kl*  Grammar: 

<LH(x,x)>::=  <PH(x)> 

<LH(y,j)>::=<PH(y)xLH(z,x)>|<LH(y,z)><PH(xb 
<Lv(x.x)>:-.=  <Pv(x)> 

<Lv(y,x)>:  =<Pv(y)><Lv(z,x)'.  |  <Lv(y,z)  xPv  (x)> 

<Ldr(x,x)>  .=  <PDr(x)'> 

<L  d  R  <y - x)>:  :=  <P  o  R  (y)>  <L  D  r  (z,x)N  I  <Ld  r  (y,z)'>  <P  D  R  (x)> 

<Ldl(x,x)>::=<Pdl(x)> 

<L D  L (y - x)>  - :  =  <P„  L  (y)>  <L D  L  (z,x)>  |  <Ld  L  (y,z)>  <PD  L  (x)> 
<R(w,h,v)>::=<LH  (v,,Vj)>  <Lv(Vj,V|,)>  <LH(vk  ,V|)> 

<Fv(vj,V|)>  /RECTANGLE 
<TI(w,h,v)>::=<LH(vi,v.)><LDR(vi,vk)> 

<LDL(vk’vi)>/  ISOSCELES  TRIANGLE 
<TR(w,h,vL::=<LD  R(v,,vk)xLv(Vj,vk)> 

<LH(vi  vi)>/  RIGHT  triangle 

<TL(w,h,v)>:  :=<Ld  L  (V(<  ,Vj)>  <LH(v|1v.)> 

<Lv(v.,vk)>/  LEFT  TRIANGLE 
<TRD(w,h,  v)>:  :  =  <Lp  L  (vj,  vk)>  <LH  (v;  ,Vj)> 

<Lv(vk.vj)>/  RIGHT  TRIANGLE  DOWN 
<TLD(w,h,v)>::=<LD  R(vk,vi)>  <LH(v,,v.)> 

<Lv(vk.V|)>/LEFT  TRIANGLE  DOWN 
<C(w.h,v)>::=<LH(V(.V|)>  <LDL(V|,vk)>  <Lv(V|,vk)> 

>LDR(vn,v|F<LH(vn,vJs<LDL(vp,vnh 
<Lv(vp,vr)xLDR(vf,v.)^/ CIRCLE 
<E(w,h,v)>::=<LH(V|.v|)xLDL(vi,vk)><Lv(v|.vk)> 

<LD  R(vm-vl^<LH(vn’Vm)><I-DL(vp'vn)'> 

<l  v<vp-vr)"<LDR(vr.vi)>/  ELLIPSE 
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2.  Transformation  Rules  for  Forming  Artificial  Lines: 


LH<vnvj)  -  Lv(Vj,vk) 

[-H(Vi,v,)  ,  Lv(v.,vk) 

LH(vi  .  Lv(vk,V|.) 

Lh <v j-v i>  -  Lv(vk’vi) 

LV<VpVi)  ,  L0L(vk,Vi) 

LV<vi-vi)  ,  LDR(v.,vk) 

LH  (vi  v  j>  '  LDR(vi-vk) 

LH(vi’Vi>  -  L ^ v j ’ vk > 

Lh(vpv,)  -  LDL(vk,Vi) 

LH(v|’Vi)  -  LDR(vk,Vi) 

LDL(vi’Vi)  -  L  v  ( v  j  > v  k  ^ 

LDr(vi-v|)  .  Lv(v.,vk) 

LDR(vj-vj)  .  Lv(V|,vk) 

LDL^vi’Vj)  ,  L  V  (v,  ,vk) 

3.  Transformation  Rules  for  the  Concatenation  of  Lines: 

L1(v1)v.),L1(v.,vk),Lm(vj,vn)  - •L'1(vPvk),(3) 

Ljl(vi’vj)’L1(vj,vk),Lm(vn,vj)  - *L  ,1(vl,vk),(3) 

4.  The  Lexicon: 

CIRCLE  :  C(ni,n2n3) 

Requirements  :  XLENGTHf  L H(vi,vj))=XLENGTH(L v  (v, , vk ))= 

XLENGTH(LH(vn>vm))=YLENGTH(Lv(vp,vr)) 
XLENGTH(LDL(v.,vk))=XLENGTH(LDR(vm  ,v,))= 

XLENGTH(LDL(Vp,vn))=XLENGTH(LDR(vr,v.))= 
YLENGTH:LDL(Vj,vk))=YLENGTH(LDR(vm,V|))= 
YLENGTH(Ldl(vp,v„))=YLENGTH(Ld  R(vr,V|)) 
Assignments  :  n,  =  CCOUNT  +  1 

n2  =  XLENGTH(LH(v|,vi))+2(XLENGTH(LDL(v.,vk))) 

n3  =  n2 

and, 

BOT  =  LH(vn,vm) 

T°P  =  L„  ( v j , v j ) 

LE  =  L  v  ( v  p ,  v  r) 

RE  =  L  y  (v  |  ,vk) 

reference  point  =  /  vk  _  vr  +-  vr,  v.  —  vn+  v  \ 

I  v  2  ) 


{(1)  ,  (2)  ,  L H ( v j , v m )  1 

1(1)  ,  (2)  ,  Ly(v(ll,vi)  / 
f  ( 1)  ,  (2)  ,  L  H  (vm  ,  Vj)  ) 
((1)  ,  (2)  ,  L  y  (  Vm  ,Vj)  j 
|(1)  ,  (2)  ,  LH(Vi,vm)  ) 
1(1)  ,  (2)  ,  Ly|V|,Vj  / 
|  (1)  ,  (2)  ,  L  H  (vm  ,  V|)  ) 
Ml)  ,  (2)  ,  Ly(v.,vm)/ 
(1),(2).  L  H  ( v  m  ,  v . ) 
(l),(2),LH(v.,vm) 
(».(2).Lh(v|,vJ 

(1)  ,  (2)  ,  Lv(vm,v.) 

(1)  ,  (2)  ,  L  H  (vm  ,  Vj) 

(1)  ,  (2)  ,  Ly(vm,Vi) 

(1)  ,  (2)  ,  LDL(vm,v.) 
(1)  ,  (2)  ,  L  D  R ( v, ,vm ) 
(1)  ,  (2)  ,  Lh  (vm  , v j ) 

(1)  ,  (2)  ,  L H (v, ,vm ) 
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ELL1PSE  E(n,,n2,n3) 

Requirements  :  XLFNCiTH(LH(v. ,V|))  TLH (vn  .vm)) 

YLENGTIHL  v  (v(  ,vk))  YLENGTU(LV<  vp  ,vr)> 
XLENGTIitLDL(v|,vk))  XLENGTIKL D  R (vm ,  v,)) 
XLENGTH(LDL(vp,v„))  XLI,.NGTH(LDR(vr,vi)) 
YLENGTH(Ldl  (v|,vll))-YLSNG.Tll(LDB(vm,vl)) 
YLENGTH(EDL(vp,vn))  Yt.ENGTIKE  D  R ( v r, v ,)) 
Assignments  :  n,  ECOIJNT  >  1 

n 2  XLENGTII(LH(v .,v jM  t  2(XLRNGT1I< L „ L ( v , . vk))» 

n  3  11 2 

and, 

BOX  EH(vn,vm) 

TOP  LH(vi,vj) 

LE  -  E  v(vp  ,vr) 

RE  -  Lv(v,,vk) 

reference  point  =  vk  _  vr  +  vr,  V|  —  vn  +  vn 
2  2 

RECTANGLE:  R(n],n2.n3) 

Requirements  XLE  NGTH(Lh(v.,v.))  =  XLENGTII(LH(vk  ,v ,)) 
YLENGTlKL  v  (Vj,V|,))  Y  LENGTIKL  v(Vj  ,V|)) 
Assignments  :  n,  RCOUNT  4  1 

n2  =  XLENGTIKL  H(v,,Vj)) 
n  3  YLENGTHfL  y(Vj.vk)) 
and, 

BOT  Lh(v,.v,) 

TOP  =  EH(vk  ,V|) 

LR  Ev(v.,vk) 

RE,  L  v  ( v  j ,  v  | ) 

reference  point  v. 

ISOSCELES  TRIANGLE  TKn,.n2,n3) 

Requirements  :  XLENGTIKL  D  r(v  j  ,vk))  XLENGTIKL  D  L  (v  k  ,V  j)) 
Assignments:  n,  TICOUNT  •  I 

n 2  XLENGTH(LH(Vi  ,v  }) 

n3  YLENGTH(LD  R(Vj,vk)) 
and, 

BOT  L  H  ( v , , v . ) 

TOP  =  vk 

LE  =  v, 

RE  -  v. 

I 

reference  point  -  v, 
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RIGUT  TRIANGl  E 
Requirements 


RIGHT  TRIANGLE  DOWN 
Requirements  : 
Assignments  : 


'.EFT  TRIANGLE 
Requirements  : 
Assignments  : 


LEFT  TRIANGLE  DOWN  : 

Requirements  . 
Assignments 


TROij.nj.nj) 

</> 

n,  IRC  Oil  NT  f  1 
n  2  XLENG  ITKL  H  (vj  ,Vj)) 
n  3  YLENCiTHIL  v(Vj,vk)) 
and, 

HOT  L  H  ( v  ■ .  v  j ) 

TOP  vk 
LE  v, 

RE  Lv(\j,vk) 
reference  point  -  v 

TRD(n,,n2,n3) 

4> 

n,  TRDCOUNT  i  l 
n2  XLENG TH(Lh(v,  ,Vj)) 
n 3  YLENGTIKL  V(vk  ,Vj)) 
and, 

HOT  vk 
TOP  =  Lh (Vj , Vj) 
l.E  v. 

RE  ^  L v (vk  ,v() 
reference  point  -  v. 

I  L(n 1 , n 2 , n 3 ) 

</> 

n, =  TLCOUNT +  I 
n2=  XLENGTIKLhIv^v,)) 
n3  YLENGTIKL  V(v,.vk)) 
and, 

BOT  1-  H  ( v  j , v  j ) 

TOP  =  vk 
LE  -  Lv(v.,vk) 

RE  =  v. 

reference  point  v, 

TLD(n 

0 

n, =  TLDCOUNT  t  I 
n2-  XLENGTH(Lh(v|,v ,)) 
n3  =  YLENGTIKL  V(vk  ,v,)) 
and, 

BOT  =  vk 
TOP  LH(v,,vi) 

LE  =  v. 

RE  Ly(vk,Vj) 
reference  point  =  v  { 
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ON  TOP  OF 

X,Y,*Rl(nl!n2nn3nn4) 

Requirement 

X( BOT)  ffl  Y (TOP) 

Assignments 

n ,  0 

n  2=  LE 

if  XCOORD(XL  E)  >  XCOORD(Yle) 
then, 

n 3  XCOORD(XL  E)  -  XCOORD(YL  E) 
n  4  RT 

if  XCOORD(XL  E)  >  XCOORD(YLE) 
then, 

n 3  =  XCOORD(YL  E)  -  XCOOR D( X L  E ) 
n  4  LT 

TO  RIGHT  OF 

X,Y,*R3  (n1in2nn3nn4) 

Requirement 

X(LE)  ffi  Y(RE) 

As  signments 

n  1  =  0 
n  2  BOT 

of  YCOORD(Xbot)  -  YCOORD(Ybot) 
then, 

n 3  YCOORD(Ybot)  -  YCOORD(Xbot) 
n  4  =  DN 

if  YCOORD(Ybot)<  YCOORD(Xbot) 
then, 

n 3  =  YCOORD(Xbot)  -  YCOORD(Ybot) 
n4=  UP 

CONTAINED  WITHIN 

X,Y,*R5(n1,n2nn3nn4,n5nn6nn7) 

Requirement 

X(l.)  ffl  Y ( 1  j) 

Assignments 

n  ,  =  0 

n2  =  LE 

n 3  =  XCOORD(Xl  e)  -  XCOORD(YL  E) 
n  4  =  RT 
n  j  =  BOT 

n6  =  YCOORD(Xbot)  -  YCOORD(Ybot) 
n7  =  UP 

5.  Transformation  Rules  for  Triplet  Set; 

Transformation  Rules  for  Isolated  Primitives 

(a,/3,*R5) ,(y  ,0,0)  - *(1),( y,/3,*R5) 

(a,/3,*Ri),(y,0,0)  - *(l),(a,y,*R5) 

(a,0,0),(/3,0,0)  - *{fi, a*  R5) 

(a,/3,*R5),(y.S,*Ri  )  - *(l),(2),(y,/3,*R5) 
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I’ran sformat inn  Rules  lor  Missing  Relations 

(a *Ri ),(<i. )  . *Ri) - *(l).<2U/3,y.*R.|> 

(«,/d,*Ri).(y,/1.*Ri) - •  ( I  ).<2).(«.)  .*Ri> 

where  i  ;  j. 

Transformation  Rules  for  Inconsistent  Relations 

(a, /d,*Ri).(/d,y.*R5).(«.  >'-*&•> - *(2>.(3) 

(^.ri.*Ri).(/),y,*R5).(y  .«.*Rt) - ♦( 2 ) . ( 3 ) 

(« ./)  ,* R5) ,(« , y  .* R i) , (/l ,<8 ,  * R i),l  5) - ►(  1>,'3).(4) 

0.u.(R5).(/i.v,*R5),<y.«.*R5) - »<2).<3) 

6.  The  Grammar  K2*  for  L*. 


sname  c  i r ^ : :  G 
name  el  lip'’::  K 
<name  reel?:.  R 
name  isos''.  T 1 
<name  rt  tn  TR 
sname  rt  tridowns; :  TRD 
•  name  1ft  tri  : :  TL 
<name  1ft  tridown  11  - D 
zero  :  0 

number::  1 12  3!4'5]6  7  8  9 

integer'-:  :=  enumber  <zero  number’  ^integer 
numtype  ::  =  <integer> 

<horizontal  dimension'’:.  -  <integer> 

<vertieal  dimension.?::  =  <integer> 

^primitive  argument  >::  =  <numtype>,<horizontal  dimension?, 

<vertical  dimension? 


<x  coord  .  <integer> 

<ycoord>::  =  Cinteger- 

<re  fp  t> : :  =  <xcoord,>,<ycoord> 

<reference  primitive  argument'-: :  =  <zcro>,  horizontal  dimension'-, 

<vertical  dimension>,<refpt> 

<  c  i  r  cl  e  > : :  -  <nante  ctr>  (  <primitive  argument’)! 

;name  cir>  (  <reference  primitive  argument) 

<el lip se> : :  =  <namc  e  1 1 ip>(<pr imi live  argument>)| 

<name  e  1 1 ip>  ( <rcference  primitive  argument;?) 
<rectangle?: :  -  <name  rect>  (  <primitive  argument>)| 

<name  reel  -preference  primitive  argument>) 
Cisosceles  triangle  =  <name  i sos?(<primiti ve  nrgument>)| 

<name  isos  >(  . reference  primitive  argument?) 

<right  triangle-  name  rt  tri  (<primitive  argument?)] 

name  rt  tri  ■(< reference  primitive  argument?) 
•-.right  triangle  down  name  rt  tridown  primitive  argument?)! 

name  rt  tridown  ■(  reference  primitive  argument?) 
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vlett  triangle  >: :  -  -  name  lft  tr  i  ■■(  .primitive  argument'') 

<name  lft  tri~>(  .reference  primitive  arguments) 

<lelt  triangle  down>::  =  uiamc  lft  tridown>(<primiti  vc  argument  )| 

vtiame  lit  tridown>(<reference  primitive  argument) 
■  primiti  ve>: :  -  <rectangle>  i  ^isosceles  triangle'll 

v  right  triangle''  |  criglit  triangle  down  >  | 

<•  le  ft  triangle>  |  <le  ft  triangle  down>  | 

<eircle>  I  <ellipse^> 

xnanie  sr>::  *R 

<msrs::  =  <integer> 
vedg  :  TO P [ BOT | HC 
shedgN::  =  LE  RLiVC 
vinsp^::  =  <intcger> 

<vdir~>: :  =  IP!  ON 
■chdir'v;  =  LTj  RT 
<vre  lpos>::  =  <vedg'>  <msp>  <vdir> 

<hrelpos''::  =  <hedg'>  <msp'>  <hdir"> 

<argvsr>::  =  <msr>,<hrelpos> 

<arghsr>::  =  <msr^,<vrelpos> 

<vertical  relation  l^::  =<name  sr>  l(<argvsr>) 

<vertical  relation  2>::=  <.name  sr>2(<argvsr>) 
horizontal  relation  3>::=  <name  sr>3(<arghsr>) 

<horizontal  relation  4  <::=  <name  srM(varghsr^) 

<vertical  relation  =  ^vertical  relation  1>| 

<vertical  relation  2> 
horizontal  relalion>::  =  <horizontal  relation  1>  | 

horizontal  relation  2> 

<directional  relation^::  =  <vertical  re  1  a tio n>  | 

horizontal  relation> 

<argcwsr>::  =  <zero>,<vrelpo s>,<hrelpo s>  | 

<zero>,<hrelpos'',<vrelpos  > 

<contained  witliin  relation  5>::=  <name  sr'>5(<argcwsr>) 

<contained  within  re lation  6 > : :  =  Cname  sr>6(<argcwsr>) 

<  contained  within  relation^::  =  Ccontained  within  relation  5>  | 

<contained  within  relation  6> 

Syntactic  relation;*::  =  Cvcrtical  relation>  | 

horizontal  relation>  | 

<  contained  within  relation> 

<hsym>: ;  =  *IISYM 
<vsym'>::  =  *VSYM 

<symf>::  =  <hsym><hdir>  |  <vsym>  <vdir> 

<rps: :  =  *RP 
<mbt>: :  =  <integer> 

<nur">::  =  <.integer> 

<rpf>: :  =  <rp>  <mbr>  <hdir>  <  nur>  | 

<rp'-  <mbr>  <vdir>  <nur'> 
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<elsv:  =  *ELS 
<rds>::  ■  *RDS 

<elrdfx:  =  -tels>  <num>  trds''  <num> 

<defs>::  =  *DEFS 
<defi>::  =  *DEF<integers 
<defsf>::  =  <defi>  <•••'>  <defs> 

<rotex:  =  *ROTE 
<rotef'>. :  -  <rote>  <in  te  gerN 

<unary  syntactic  function^: ;  =  <symf>  |  <rpf>  |  <elrdf>  I  •Crotef'' 

''binary  syntactic  functions :  =  <defsf' 

<sss>::  =  <primi  tiv  exp  rim  i  ti  v  e>  <  syntactic  relation>  | 
<sss><primiti ve>< syntactic  relation>  | 
cprimiti ve>  <sss>  <syn tactic  re  1  a tion>  | 
<sss>  <s  ss>  <syntactic  relation>  | 
<sss><unarv  syntactic  function'' | 

DEFi  <SSSs  DEFS 
<SSX  =  <S  S  S'' 

7.  1  he  First  Level  of  the  Semantic  Component  Grammar  K3* 

<figure>::  =  <house  typo  |  <silo  type>  | 

<store  type'-  |  <tree  type'  | 

<bulbous  type>  j  Coffice  type> 

<house  type>:  .  =  <doghouse'>  |  <shed>  |  vhouse>  | 

<church^  |  <garage  >  |  <barn>  | 

<school>  |  <barbershop> 

<silo  type  =  <silo  type>  |  <lighthouso  | 

<bouys  |  <tower' 

<tree  typo::  =  <tree>  |  Ctent^  \  <radio'>  |  <tower>- 
<store  type'>::  =  <store> 

<office  type  >::  =  coffice  bldg^- 1  < 1 1  a g  pole> 

<bulbous  type'': :  =  <water  tower>  |  <barber  pole>  | 

<tree^  |  <lamppost^  |  <sign> 

<shed> : :  -  <faeade  1>/(1, 1),(1 ,2), (2, 2), 

(2, 3), (3, 2), (3, 3) 

<garage>  ::  =  <facade  l>/(  1 , 1 ),( 1 ,2), (2, 2), 

(2, 3), (3, 2), (3, 3) 

<doghouse  >::  =  <facude  l>/(  1 . 1),(1 ,2), (2, 2), 

(2,3),(3,2),(3,3) 

<house>::  =  <facade  |  <facade  2>  (1 , 1),(1 ,2), (2, 2), 

(2, 3), (3, 2), (3, 3) 

,school>::  =  <facade  l'*  |  <facade  2>/(l,l),(l  ,2). (2, 2), 

(2, 3), (3, 2), (3, 3) 

<barbershop> . :  =  <facade  1>  |  ^facade  2>/(l,  1),(1 ,2), (2, 2), 

(2, 3), (3, 2), (3, 3) 
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barn <faca«le  1  |  <facade  3> .  (4, 4), (4, 10), (6, 13), 

(7. 7) , (7, 8), (8, 7), 

(8. 8) , (9, 6), (9, 9), 

(10, 9), (10, 10), (13, 13) 

<church>::  <cros sxfacade  4><vertical 


relation  1  1 , 1 ) ,  ( 1 ,2), 

(1.3) , (2,1), (2, 2), (2, 4), 

(3. 2) , (3, 3), (4, 4), (4, 10), 

(5. 4) , (5, 6), (6, 5), (7 ,7), 

(7. 8) , (8, 7), (8, 8), (9, 6), 

(9. 9) , (10, 9), (10, 10), 

(11, 10), (11, 12), (11, 12), 

(12, 10), (13, 13) 

<s ilo> : :  =  <facade  0>  (1 ,2), (1 ,3), (3,1), (3, 3) 

<lighthouse>-.:  =  <facade  0>  |  <roof>  <facade  5>  <vertieal 

relation  l>/(  1,1), 

(1.2) , (1,3), (2,1), 

(2. 2) , (2, 3), (3,1), 

(3. 2) , (3, 3) 

<to\ver>::  =  <faeade  0>  |  <roofxfacade  5>  Cveitical 

relation  l>/(  1 , 1), 

(1.2) , (1,3), (2,1), 

(2. 2) , (2, 3), (3.1), 

(3. 2) , (3, 3) 

<buoy>::  =  <facade  0>  /  (1,1), (1,3), (3,1), (3, 3) 

<tree>::  =  <facade  0>  |  <facade  6>  |  <facade  7>/(  1 ,1),(1 ,2), (2 , 1 ), 

(2. 2) , (2, 3), (2, 4), 

(3. 2) , (4,2), (4, 4) 

Ctent  >::  =  <isosceles  tri angle  >  <isosce!e s  triangle> 

<contained  within  relation  5> 

<radio  tower>::=  <facade  0>/ ( 1 , 1) ,( 1 , 2) 

<store>: :  =  <front  3>/(l,l),(2,l),(2,2) 

Coffice  bldg>::=  <front  5  >  <front  3><vertical  relation  1>  | 

<front  5><office  bldg>  <vertical 

relation  1  >/(5 , 6)  ,(6,5) 
<flag  pole>::=  <;flag>  <facade  6>  horizontal 

relation  3> / (2 , 3 ) , (2 ,4) , 

(3. 2) , (3, 3), (4, 2) 

<water  tower>::=  <facade  6>/(l,2),(2,l),(2,2) 

<barber  pole>::=  <facade  6>/(l,2),(2,l),(2,2) 

<lamppost>: :  =  <facade  6>/(l,2),(2,l),(2,2) 

<sign>::  =  <facade  6>/(l,2),(2,l),(2,2) 

<facade  0>:;  =  <roof>  <pane  1>  <vertical  relation  1> 

<facade  1>::  =  <roof> <front  lxvertical  relation  1> 

<facade  2> : :  =  <roof><front  2><vertical  relation  1> 

<facade  3>::  =  <loft>  <roofx.contained  within  relation  5> 

<front  3  >  <vertical  relation  1> 
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Ct'acade  4x  =  ^.steeple'*  <.front  4'xvertical  relation  1 
•cfacade  5x.  -  <paneb  <base>  < v ertic al  relation  1> 

<faeade  6'  ::  =  <bulb  1>  <panel  >  <vertical  relation  l> 

^facade  7'>  .  :=  <  bulb  2>  -pane  l'-  <v  ertic  al  relation  1 

<inside  lx.  =  <window^  j  <door>  I 

<window>  <door>  chorizontal  relation  3> 
<inside  1  <window>  <horizontal  relation  3> 

.inside  2>::=  <double  door>  | 

<windowxdouble  doorXverti c al  relation  1> 
<inside  3x  =  <windowXwindo  wxdirectional  relationa  l 

<windo  w">  <in side  3~>  -  directional  relation'- 
.front  lx  =  <panel>  |  <door>  <panel">  ^contained  within 

relation  5> 

front  2x.  -•=  < front  1  ~>  |  <inside  1>  <panelxcontained  within 

relation  ;”> 

from  3 N ;  =  front  2'>|<inside  2> •  paneb <contained  within 

relation  5> 

front  4 x.  cfront  3>  |  <right  wingxfront  3> 

<horizontal  relation  3xleft  wing> 

^horizontal  relation  3> 

<front  5>:;  =  <inside  3><pane!  ">  <contained  within 

relation  5^  |  <panel~> 

<steeple>::  =  <roof>  |  <stained  glass>  <roof>  <contained  within 

relation  5> 

<stained  glass>::  =  <isosceles  triangle> 

<roofx  =  <isosceles  triangle> 

<panelx  =  <rectangle> 

<cross>::=  <right  arm> <upright> <horizontal  relation  3 
<1  eft  arm>  <horizontal  relation  3> 

<right  arm > ::  =  <rectangle> 

<upr  i  gh  t> : :  =  <rectangle> 

<1  eft  arm> : :  =  <rectangle> 

< left  wingX  =  <right  triangle>  <rectangleXvertical 

relation  1> 

< right  wingx.  =  <1  eft  triangle>  <rectangle>  <vertical 

relation  1> 

<window>::=  <rectangle> 

<double  doorx  =  <left  doorXright  doorXhorizontal  relation  3> 
<doorsX.  =  <rectangle> 

<1  eft  door> : :  =  <rectangle> 

<ri ght  door>::  =  <rectangle> 

<flagx  =  <rectangle> 
cbulb  1>::  =  <circle> 

<bulb  2> : :  =  <ellipse> 

<base>::=  <rectangle> 

<loft> : :  =  <rectangle> 
cellipsex  =  E(n,h,v) 
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<circle> : .  =  C(n,h,v) 
<rectangle>: :  =  R(n,h,v) 
<isosceles  triangle>::=  TI(n,h,v) 
<right  triangle^: :  =  TR(n,h,v) 
<right  triangle  down'-:^  TRD(n,h,v) 
<left  triangle>::=  TL(n,h.v) 

<  l e 1 1  triangle  down>::=  TLD(n,h,v) 
■cvertical  relation  1>::=  *Rl(n,,n2) 
<horizontal  relation  3>: :=  *R3(nj,n2) 
<eontained  within  relation  5>::  =  *R5(n  ,,n2,n3) 


*R5(O.BOT  VC) 


SII  >H(3>  2 \\ (  3 ) 


C  hurch  (CH) 
Barn  (BN) 


right  wing 


Church  (CH) 
Barn  (BS) 
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R3(0  BOT) 


I 

I 

I 

i 

I 

I 

! 

I 

I 

i 

1 

i 

I 

1 

I 

I 

i 


Bony (BY) 
Tow  er  (TW ) 


iKD  \\(D  nv  >\\d)  wo  ;  >\vc)  w<3) 
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Free  (TR) 


116 


118' 
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8.  The  Second  Level  of  ihe  Semantic  Component  Grammar  K3* 

<scene'>::  ccitys  i  ccountys  |  shorn e.  backyard  '  j  <farm  > 
sschool  yard • !  church  yards | 
sbarbers '  sforest  cfields  |  < camps  ( 
<shopping  centers 

city  ::  <barbers  -^shopping  center  -  chorizontal  relations 
vhomes  Csehool  ' ^horizontal  relations  j 
vschools  schurchs  chorizontal  relation- 
Ccountry  ■>::  =  chome  •  cfarms  chorizontal  relation 

homes::  =  <house  ■  backyards  chorizontal  relations  , 
chouses  <garageN  <horizontal  relation 
chou ses  -  doghouse  chorizontal  relation 
strees  homes  chorizontal  relation  ■ 
shouses  '  lampposts  chorizontal  relation  ■  1 . 1 ), 

(1.2) ,  (1,3).  (1,4),  (1,5), 

(2. 2) .  (2, 4),  (3, 3),  (3. 4), 
(3,6) 

ebackyard  -  <garage>  cdoghouses  ^.horizontal  relation  | 

'  doghouses  sslied'  Chorizontal  relation 
stree  •  cbackyard>  <horizontal  relation  -  (2,2), 

(2. 3) , (2, 4). (3, 3), 

(3. 4) , (3, 6) 

cfarms::  =  <farms ssheds  <horizontal  relations  | 

<farms  <silos  <horizontal  relations  j 
cfarms  csheds  chorizontal  relations 
< t re e>  < farms  chorizontal  relations/ <  1 , 2) ,{ 1 ,3) 
cschool  yards::  =  cschools  cflag  poles  chorizontal  relations 
Cchurch  v  ard> : .  =  Cchurchs  c  trees  chorizontal  re  la  lions  | 

ctrees  cchurch  yards  chorizontal  relations 
cbarberss:;=  Cbarbershop  •  Cbarber  poles  chorizontal 

relations/G  .2) 

Cforests-. :  =  Ctrees  ctrees  Chorizontal  relations  | 

ctrees  cforests  chorizontal  relations 
cfield  ::  =  Cwater  towers  c  forests  chorizontal  relations  | 

Ctowers  cforests  chorizontal  relations  | 
cradio  to wers  cf  ores ts  Chorizontal 

relations/(l  ,2). 

(1.3) , (1,4), (2, 3), 

(2.4) 

Ccamps::  =  c tents  < tents  Chorizontal  relations  | 

Ctents  Ccamps  chorizontal  relations 
cshopping  centers  ::  =  C stores  Cs igns  Chori zonlal  rela tions  | 

CstoreS  Cstores  Chorizontal  relations  | 

cstores  Cshopping  centers  Chorizontal  relations  | 

cshopping  centers  cshopping  center  ' 

Chorizontal  relations/(  1 ,2) 


Home  HIM) 
Backyard  ( B^  ) 
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•  arm  ( F\1) 


121 


barber  pol 


Held(FD) 


311(1) 
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